Agent Crawl Readiness

io.github.sadri-dridiv1.0.0Updated Sep 23, 2026

Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage.

VerifiedStreamable HTTPWeb executableWeb Search & ScrapingAI & ML

Installation

In SourceWeft

  1. Open Agent Crawl Readiness in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Web executable via Streamable HTTP. Remote servers run from the web runtime once configured in a workspace.

Other MCP clients

Add this to your client's mcpServers config.

{
  "mcpServers": {
    "agent-crawl-readiness": {
      "type": "http",
      "url": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"
    }
  }
}

Tools

0
Tool metadata has not been indexed yet.

Version history

1
  1. v1.0.0LatestSep 23, 2026