Robots.txt & AI Crawler Checker

dev.workers.quietworklabs.robots-checkPublished by robots-check.quietworklabs.workers.devOfficial publisherv1.0.0Updated Oct 25, 2026

robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

VerifiedStreamable HTTPWeb executableWeb Search & Scraping

Overview

Robots.txt & AI Crawler Checker's hosted MCP server (robots-check.quietworklabs.workers.dev), published by robots-check.quietworklabs.workers.dev; no sign-in required; Streamable HTTP; 3 tools.

No AI overview yet: these facts come from the registry entry and the server itself.

Publisher
robots-check.quietworklabs.workers.dev
Endpoint
robots-check.quietworklabs.workers.dev
Last checked

Installation

In SourceWeft

  1. Open Robots.txt & AI Crawler Checker in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Web executable via Streamable HTTP. Remote servers run from the web runtime once configured in a workspace.

Use with an agent

Paste this prompt into Claude Code, Codex, Cursor or another agent — or into SourceWeft chat. The agent reads this server's install guide, shows you its publisher, endpoint and sign-in, and adds it to your client once you agree. Keys never go into a command.

Read https://sourceweft.com/mcp/dev.workers.quietworklabs.robots-check%2Frobots-check/install.md and connect the MCP server it describes to my client. Before changing any config, show me its publisher, endpoint and sign-in method and wait for my OK. Never put an API key in a command or in a file I would commit; ask me where it should come from.

Read the install guide the agent follows

Other MCP clients

Add this to your client's mcpServers config.

{
  "mcpServers": {
    "robots-txt-ai-crawler-checker": {
      "type": "http",
      "url": "https://robots-check.quietworklabs.workers.dev/mcp"
    }
  }
}

Tools

3

robots_check

May this user-agent crawl this URL? (RFC 9309)

Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else *, longest match, Allow wins ties, * and $ wildcards, 4xx=allow all, 5xx=disallow all). Returns allowed, the deciding rule and line, crawl-delay and sitemaps. Deterministic, no AI. Price: $0.001 USDC per call via x402 (Base); the first 5 calls per day per client are free.

Parametersurlrequireduser_agent

ai_crawler_access

Which AI crawlers a site allows or blocks

AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more) grouped by training/search/user purpose, plus page meta robots, X-Robots-Tag (noindex/noai) and llms.txt presence. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.

Parametersurlrequired

sitemap_check

Find and validate an XML sitemap

Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 50,000 limit, lastmod format and range, relative/off-host/duplicate/http URLs, gzip support. Returns sample URLs or child sitemaps. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.

Parametersurlrequired

Version history

1
  1. v1.0.0LatestOct 18, 2026

Score

72% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 daysRequired25 pts
  • Auth declaredRequired10 pts
  • Tools listedRequired15 pts
  • DescriptionRequired15 pts
  • Logo10 pts

    Publish an icon in the registry entry.

  • Documentation15 pts

    Declare a GitHub repository with a README, or serve resource_documentation in the OAuth protected-resource metadata.

  • LicenseNot applicable

    A hosted service is used under its provider's terms, so no license is expected here. This check is left out of the percentage.