Robotstxt Ai

io.github.sharozdawaPublished by GitHub account sharozdawav1.0.1Updated Oct 24, 2026

Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.

VerifiedSTDIODesktop onlyWeb Search & ScrapingDeveloper Tools

Overview

AI-generated overview

Analyzes and generates robots.txt rules for AI crawlers, letting an assistant check which of 25+ bots are blocked on a site and draft new rules.

What you can do
  • Fetch and analyze a site's robots.txt from a URL, or analyze pasted robots.txt content, to see which AI bots are blocked
  • Generate a robots.txt with chosen blocked bots and custom rules
  • List known AI bots with user-agents, companies and descriptions
  • Check whether a specific bot is blocked on a given website
When to use it
Useful when you want an assistant to audit a site's crawler policy, see whether GPTBot, ClaudeBot, PerplexityBot or similar bots are allowed, or draft robots.txt rules without editing the file by hand.
Requirements
Runs as a local stdio process, typically launched with npx from the npm package robotstxt-ai-mcp. No accounts, API keys or environment variables are declared. Network access is needed to fetch robots.txt files from remote sites.
Before you install
Fetching robots.txt sends requests to the URLs you provide, so avoid pointing it at internal or private hosts. Generated rules are drafts; review them before publishing, since they change how crawlers may access a site.

Installation

In SourceWeft

  1. Open Robotstxt Ai in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Desktop only via STDIO. STDIO servers start a local process, so they need the SourceWeft desktop host.

Other MCP clients

Follow the launch instructions in the repository.

README

[Image]

[npm] [GitHub stars] [License] [robotstxt-ai MCP server]

Manage which AI bots can crawl your website — visually.

Toggle GPTBot, ClaudeBot, PerplexityBot, and 20+ AI crawlers on/off with a simple UI. Analyze any site's robots.txt instantly.

Features

  • Visual Toggle UI — Block or allow AI bots with simple on/off switches
  • 20+ AI Bots Database — GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent, and more
  • Analyze Existing robots.txt — Paste or fetch any robots.txt to see which AI bots are blocked
  • Generate robots.txt — Create a complete robots.txt with your chosen rules
  • MCP Server — Use with Claude Desktop, Cursor, or any MCP-compatible AI assistant
  • Check Bot Status — Verify if a specific bot is blocked on any website

MCP Tools

ToolDescription
fetch_robotsFetch and analyze a robots.txt from any URL
analyze_robotsAnalyze pasted robots.txt content for AI bot blocking status
generate_robotsGenerate a robots.txt with specified blocked bots and custom rules
list_ai_botsList all known AI bots with user-agents, companies, and descriptions
check_bot_statusCheck if a specific bot is blocked on a given website

Installation

Web App

bash
git clone https://github.com/sharozdawa/robotstxt-ai.gitcd robotstxt-ainpm installnpm run dev

Open http://localhost:3000 in your browser.

MCP Server — Claude Desktop

Add to your claude_desktop_config.json:

json
{  "mcpServers": {    "robotstxt-ai": {      "command": "npx",      "args": ["-y", "robotstxt-ai-mcp"]    }  }}

MCP Server — Cursor

Add to .cursor/mcp.json:

json
{  "mcpServers": {    "robotstxt-ai": {      "command": "npx",      "args": ["-y", "robotstxt-ai-mcp"]    }  }}

Tracked Bots

The server knows about 25+ bots including:

  • AI Crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent
  • AI Search: ChatGPT-User, OAI-SearchBot, PerplexityBot, YouBot
  • Search Engines: Googlebot, Bingbot, YandexBot, Baiduspider, DuckDuckBot

Why robotstxt.ai vs Manual Editing

Featurerobotstxt.aiManual Editing
Visual toggle UIYesNo
20+ AI bots databaseYesResearch yourself
Analyze existing robots.txtYesNo
MCP ServerYesNo
PriceFreeFree but tedious

More Open Source SEO Tools

ToolDescription
awesome-seo-mcp-serversCurated list of SEO MCP servers and agent skills
indexnow-mcpInstant URL indexing via IndexNow
schema-genSchema.org JSON-LD markup generator
ai-visibilityAI brand visibility tracker

License

MIT

Built by Sharoz Dawa — SEO Professional & Digital Marketing Expert

[GitHub]

[Image]

Source: README.md at commit 0df01d5

Tools

0
SourceWeft has not started this package in its sandbox yet, so its tools are not known.

Version history

1
  1. v1.0.1LatestSep 16, 2026

Score

50% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 days25 pts

    Keep the endpoint answering the MCP handshake; SourceWeft probes it every 30 days.

  • Auth declared10 pts

    Declare the credential headers in the registry entry, or keep the endpoint answering the probe so its auth can be classified.

  • Tools listed15 pts

    Answer tools/list without a credential, or ship a package that starts in a sandbox without one.

  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts