Ai Crawler Readiness

com.lumnikaPublished by lumnika.comOfficial publisherv1.0.0Updated Oct 24, 2026

Can AI crawlers read this site? GPTBot/ClaudeBot verdicts, robots vs edge blocking, 47k-domain index

VerifiedStreamable HTTPWeb executableWeb Search & ScrapingSecurity & Monitoring

Overview

AI-generated overview

Remote MCP endpoint that reports whether AI crawlers such as GPTBot and ClaudeBot can read a site, comparing robots rules with edge blocking.

What you can do
  • Check whether AI crawlers can read a given site, with verdicts for crawlers such as GPTBot and ClaudeBot.
  • Compare robots.txt rules against edge-level blocking to see where access is actually refused.
  • Draw on a 47k-domain index for context on crawler access.
  • No individual tools are listed in the manifest, so exact operations are not documented.
When to use it
Useful when you want an assistant to assess whether a site is readable by AI crawlers, for example before publishing content for AI discovery or when diagnosing why a crawler is blocked.
Requirements
A remote streamable HTTP endpoint at shop.lumnika.com; OAuth authentication is required. No environment variables or headers are declared.
Before you install
Access requires OAuth sign-in to the provider's service, so an account and its authorization are involved. The server is hosted by a third party and any site you query is submitted to it; review the provider's terms before use.

Installation

In SourceWeft

  1. Open Ai Crawler Readiness in the dashboard and add it to a workspace.
  2. Connect the required credentials. They are stored privately and never shown on public pages.
  3. Enable the server for the chats that should use its tools.

Web executable via Streamable HTTP. Remote servers run from the web runtime once configured in a workspace.

Use with an agent

Paste this prompt into Claude Code, Codex, Cursor or another agent — or into SourceWeft chat. The agent reads this server's install guide, shows you its publisher, endpoint and sign-in, and adds it to your client once you agree. Keys never go into a command.

Read https://sourceweft.com/mcp/com.lumnika%2Fai-crawler-readiness/install.md and connect the MCP server it describes to my client. Before changing any config, show me its publisher, endpoint and sign-in method and wait for my OK. Never put an API key in a command or in a file I would commit; ask me where it should come from.

Read the install guide the agent follows

Other MCP clients

Add this to your client's mcpServers config. The client completes OAuth sign-in on first connect.

{
  "mcpServers": {
    "ai-crawler-readiness": {
      "type": "http",
      "url": "https://shop.lumnika.com/ai-readiness/mcp"
    }
  }
}

Tools

0
This server asks for a credential such as an API key, so SourceWeft could not read its tools without one and has not indexed them.

Version history

1
  1. v1.0.0LatestSep 20, 2026

Score

67% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 daysRequired25 pts
  • Auth declaredRequired10 pts
  • Tools listed15 pts

    Answer tools/list without a credential, or ship a package that starts in a sandbox without one.

  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts

    Declare a GitHub repository with a README, or serve resource_documentation in the OAuth protected-resource metadata.

  • LicenseNot applicable

    A hosted service is used under its provider's terms, so no license is expected here. This check is left out of the percentage.