Robots Txt Ai Crawler Audit

Local MCP server that audits a robots.txt file line by line and explains what each AI crawler token actually controls.

What you can do

• Audit a robots.txt file line by line, checking each directive against AI crawler tokens.
• Explain what each AI crawler token actually controls, such as training versus other AI uses.
• Report findings so an assistant can discuss which crawlers are allowed or blocked.

When to use it

Useful when you want to know whether a site's robots.txt permits or blocks AI crawlers, for example before publishing content or reviewing crawler policy.

Requirements

Runs as a local process on the user's machine via npx from the npm package; desktop only, no web version. No authentication, environment variables or headers are declared.