dev.workers.mike-tusa.robotsgatePublished by robotsgate.mike-tusa.workers.devOfficial publisherv0.3.1Updated Oct 24, 2026
Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.
Lets an assistant generate, validate and check robots.txt rules aimed at AI crawlers.
Web executable via Streamable HTTP. Remote servers run from the web runtime once configured in a workspace.
Paste this prompt into Claude Code, Codex, Cursor or another agent — or into SourceWeft chat. The agent reads this server's install guide, shows you its publisher, endpoint and sign-in, and adds it to your client once you agree. Keys never go into a command.
Read https://sourceweft.com/mcp/dev.workers.mike-tusa.robotsgate%2Frobotsgate/install.md and connect the MCP server it describes to my client. Before changing any config, show me its publisher, endpoint and sign-in method and wait for my OK. Never put an API key in a command or in a file I would commit; ask me where it should come from.Add this to your client's mcpServers config.
{
"mcpServers": {
"robotsgate": {
"type": "http",
"url": "https://robotsgate.mike-tusa.workers.dev/mcp"
}
}
}List the AI crawler registry
Return RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category (training, dataset, search, user_fetch or other), whether the operator says it respects robots.txt, and a link to the operator's documentation, plus unverified entries and notes. Read-only, no network access. Same data as GET /api/crawlers.
categoryGenerate a robots.txt for AI crawlers
Build a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token overrides, a default policy for other bots, paths, sitemaps and custom rules. Returns the file text, a per-crawler summary and notes. Read-only: it only returns text and changes nothing. Same engine as POST /api/generate.
dateagentsdefaultsitemapsitemapscategoriesallow_pathscustom_rulesdisallow_pathsValidate robots.txt text
Validate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or blocked and by which rule. Read-only, no network access. Same engine as POST /api/validate. Up to 50,000 characters over MCP (the whole JSON-RPC request must also fit in 64 KB); for larger files use POST /api/validate or check_site.
pathrobots_txtrequiredCheck which AI crawlers a live site allows
Fetch a site's /robots.txt (http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes; results cached for 5 minutes) and report, for one path, which registry AI crawlers are allowed or blocked, plus validation errors and warnings. Read-only: one GET of /robots.txt with a RobotsGate user-agent. Same engine as GET /api/check.
urlrequiredpath72% of the applicable checks met
How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.
Publish an icon in the registry entry.
Declare a GitHub repository with a README, or serve resource_documentation in the OAuth protected-resource metadata.
A hosted service is used under its provider's terms, so no license is expected here. This check is left out of the percentage.
