Robots Txt Checker

io.github.ryuamarinesPublished by GitHub account ryuamarinesv1.0.93058Updated Oct 29, 2026

Checks if a URL path is allowed by robots.txt. x402 payment required (Base mainnet USDC).

VerifiedStreamable HTTPWeb executableWeb Search & Scraping

Overview

AI-generated overview

Remote MCP server with one tool that checks whether a URL path is allowed by a site's robots.txt, paid per call via x402 in USDC on Base mainnet.

What you can do
  • Check robots.txt rules: given a URL path, report whether the site's robots.txt allows it.
  • The single tool, robots_txt_checker, covers this one lookup; no other tools are exposed.
When to use it
Useful when an assistant needs a quick robots.txt allow/disallow verdict for a specific URL path, for example before crawling or fetching a page, without implementing the parsing itself.
Requirements
Remote streamable HTTP endpoint at the provider's Cloudflare Workers host; no package, environment variable or header is declared. Calls require x402 payment in USDC on Base mainnet, so a compatible paying client and funded wallet are needed.
Before you install
Each call is a paid x402 transaction in USDC on Base mainnet, so usage spends real funds; the server is unauthenticated apart from payment, and the URL paths you submit are sent to a third-party endpoint.

Installation

In SourceWeft

  1. Open Robots Txt Checker in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Web executable via Streamable HTTP. Remote servers run from the web runtime once configured in a workspace.

Use with an agent

Paste this prompt into Claude Code, Codex, Cursor or another agent — or into SourceWeft chat. The agent reads this server's install guide, shows you its publisher, endpoint and sign-in, and adds it to your client once you agree. Keys never go into a command.

Read https://sourceweft.com/mcp/io.github.ryuamarines%2Frobots-txt-checker/install.md and connect the MCP server it describes to my client. Before changing any config, show me its publisher, endpoint and sign-in method and wait for my OK. Never put an API key in a command or in a file I would commit; ask me where it should come from.

Read the install guide the agent follows

Other MCP clients

Add this to your client's mcpServers config.

{
  "mcpServers": {
    "robots-txt-checker": {
      "type": "http",
      "url": "https://ai-factory-mcp.ryuamarines.workers.dev/robots-txt-checker/mcp"
    }
  }
}

README

This repository has no README.

Tools

1

robots_txt_checker

Checks if a URL path is allowed by robots.txt. x402 payment required (Base mainnet USDC).

Parameterschecksrequiredpayment

Version history

2
  1. v1.0.93058LatestOct 29, 2026
  2. v1.0.93057Sep 16, 2026

Score

75% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 daysRequired25 pts
  • Auth declaredRequired10 pts
  • Tools listedRequired15 pts
  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts

    Declare a GitHub repository with a README, or serve resource_documentation in the OAuth protected-resource metadata.

  • License10 pts

    Declare a license in the repository or the registry entry.