dev.workers.mike-tusa.robotsgate由 robotsgate.mike-tusa.workers.dev 發布官方發布者v0.3.1更新於 Oct 24, 2026
Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.
讓助理產生、驗證與檢查針對 AI 爬蟲的 robots.txt 規則。
Web executable，透過 Streamable HTTP。 遠端服務在工作區中設定後即可從網頁執行環境執行。
把這段提示詞貼上到 Claude Code、Codex、Cursor 等代理程式，或 SourceWeft 對話中。代理程式會讀取該伺服器的安裝指南，先向你展示發布者、端點和登入方式，經你同意後再把它新增到你的客戶端。金鑰不會寫進任何命令。
閱讀 https://sourceweft.com/mcp/dev.workers.mike-tusa.robotsgate%2Frobotsgate/install.md，並按其中說明把這個 MCP 伺服器接入我的客戶端。修改任何設定之前，先告訴我它的發布者、端點和登入方式，等我確認。不要把 API 金鑰寫進命令或我會提交的檔案；問我金鑰應從哪裡取得。把它新增到你客戶端的 mcpServers 設定中。
{
"mcpServers": {
"robotsgate": {
"type": "http",
"url": "https://robotsgate.mike-tusa.workers.dev/mcp"
}
}
}List the AI crawler registry
Return RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category (training, dataset, search, user_fetch or other), whether the operator says it respects robots.txt, and a link to the operator's documentation, plus unverified entries and notes. Read-only, no network access. Same data as GET /api/crawlers.
categoryGenerate a robots.txt for AI crawlers
Build a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token overrides, a default policy for other bots, paths, sitemaps and custom rules. Returns the file text, a per-crawler summary and notes. Read-only: it only returns text and changes nothing. Same engine as POST /api/generate.
dateagentsdefaultsitemapsitemapscategoriesallow_pathscustom_rulesdisallow_pathsValidate robots.txt text
Validate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or blocked and by which rule. Read-only, no network access. Same engine as POST /api/validate. Up to 50,000 characters over MCP (the whole JSON-RPC request must also fit in 64 KB); for larger files use POST /api/validate or check_site.
pathrobots_txt必填Check which AI crawlers a live site allows
Fetch a site's /robots.txt (http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes; results cached for 5 minutes) and report, for one path, which registry AI crawlers are allowed or blocked, plus validation errors and warnings. Read-only: one GET of /robots.txt with a RobotsGate user-agent. Same engine as GET /api/check.
url必填path已滿足適用檢查項的 72%
該評分衡量此收錄條目的完整程度，依據註冊中心條目、端點與公開來源計算得出。它不是對伺服器本身的評價，也不表示伺服器安全或目前可用：可連通性是帶日期的檢查結果，評分也不包含安裝量或使用者評價。
請在註冊中心條目中發布圖示。
請宣告包含 README 的 GitHub 儲存庫，或在 OAuth 受保護資源後設資料中提供 resource_documentation。
託管服務按其提供方的條款使用，因此這裡不要求授權條款。此檢查項不計入百分比。
