dev.workers.mike-tusa.robotsgate由 robotsgate.mike-tusa.workers.dev 发布官方发布者v0.3.1更新于 Oct 24, 2026
Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.
让助手生成、校验和检查针对 AI 爬虫的 robots.txt 规则。
Web executable，通过 Streamable HTTP。 远程服务在工作区中配置后即可从网页运行时运行。
把这段提示词粘贴到 Claude Code、Codex、Cursor 等智能体，或 SourceWeft 对话中。智能体会读取该服务器的安装指南，先向你展示发布者、端点和登录方式，经你同意后再把它添加到你的客户端。密钥不会写进任何命令。
阅读 https://sourceweft.com/mcp/dev.workers.mike-tusa.robotsgate%2Frobotsgate/install.md，并按其中说明把这个 MCP 服务器接入我的客户端。修改任何配置之前，先告诉我它的发布者、端点和登录方式，等我确认。不要把 API 密钥写进命令或我会提交的文件；问我密钥应从哪里获取。把它添加到你客户端的 mcpServers 配置中。
{
"mcpServers": {
"robotsgate": {
"type": "http",
"url": "https://robotsgate.mike-tusa.workers.dev/mcp"
}
}
}List the AI crawler registry
Return RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category (training, dataset, search, user_fetch or other), whether the operator says it respects robots.txt, and a link to the operator's documentation, plus unverified entries and notes. Read-only, no network access. Same data as GET /api/crawlers.
categoryGenerate a robots.txt for AI crawlers
Build a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token overrides, a default policy for other bots, paths, sitemaps and custom rules. Returns the file text, a per-crawler summary and notes. Read-only: it only returns text and changes nothing. Same engine as POST /api/generate.
dateagentsdefaultsitemapsitemapscategoriesallow_pathscustom_rulesdisallow_pathsValidate robots.txt text
Validate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or blocked and by which rule. Read-only, no network access. Same engine as POST /api/validate. Up to 50,000 characters over MCP (the whole JSON-RPC request must also fit in 64 KB); for larger files use POST /api/validate or check_site.
pathrobots_txt必填Check which AI crawlers a live site allows
Fetch a site's /robots.txt (http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes; results cached for 5 minutes) and report, for one path, which registry AI crawlers are allowed or blocked, plus validation errors and warnings. Read-only: one GET of /robots.txt with a RobotsGate user-agent. Same engine as GET /api/check.
url必填path已满足适用检查项的 72%
该评分衡量此收录条目的完整程度，依据注册中心条目、端点与公开来源计算得出。它不是对服务器本身的评价，也不表示服务器安全或当前可用：可连通性是带日期的检查结果，评分也不包含安装量或用户评价。
请在注册中心条目中发布图标。
请声明包含 README 的 GitHub 仓库，或在 OAuth 受保护资源元数据中提供 resource_documentation。
托管服务按其提供方的条款使用，因此这里不要求许可证。此检查项不计入百分比。
