Crawler IP Verifier - real Googlebot or fake

dev.workers.pathwren.wwwPublished by www.pathwren.workers.devOfficial publisherv1.4.0Updated Oct 24, 2026

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key

The publisher's own server: Agent Discovery Doctor - llms.txt, agent card, mcp.json

VerifiedStreamable HTTPWeb executableWeb Search & ScrapingSecurity & Monitoring

Overview

AI-generated overview

A remote, keyless MCP endpoint that checks IP addresses against mirrored crawler-operator prefix lists to tell real Googlebot-style crawlers from fakes.

What you can do
Report which crawler-operator IP range lists the host mirrors and their freshness and sizing statistics, including prefixes shared by several operators. Check a single address, or the caller's own address, against every mirrored operator prefix list. Check up to 500 addresses in one batch, returning the matching prefix per address. Look up which published operator prefixes contain, sit inside, or overlap a given CIDR. Export operator ranges as ready-to-paste ACL configs such as nginx, Apache, HAProxy or Cloudflare firewall rules. Show per-crawler verification methods, including whether an operator publishes ranges or documents reverse DNS.
When to use it
Useful when you need to decide whether a visitor claiming to be a search crawler is genuine, to audit or build IP allow/deny lists for crawler traffic, or to size and refresh crawler range data before writing firewall rules.
Requirements
A remote streamable HTTP endpoint; no package, API key, account or headers are declared. Network access to the provider's host is required.
Before you install
The caller's own address is reported by the whoami and verify tools, so the endpoint sees the connecting IP. Results depend on the mirrored operator lists and their freshness, so treat verdicts as advisory. Exported ACL configs should be reviewed before being applied to live firewalls.

Installation

In SourceWeft

  1. Open Crawler IP Verifier - real Googlebot or fake in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Web executable via Streamable HTTP. Remote servers run from the web runtime once configured in a workspace.

Use with an agent

Paste this prompt into Claude Code, Codex, Cursor or another agent — or into SourceWeft chat. The agent reads this server's install guide, shows you its publisher, endpoint and sign-in, and adds it to your client once you agree. Keys never go into a command.

Read https://sourceweft.com/mcp/dev.workers.pathwren.www%2Fcrawler-ip-verifier/install.md and connect the MCP server it describes to my client. Before changing any config, show me its publisher, endpoint and sign-in method and wait for my OK. Never put an API key in a command or in a file I would commit; ask me where it should come from.

Read the install guide the agent follows

Other MCP clients

Add this to your client's mcpServers config.

{
  "mcpServers": {
    "crawler-ip-verifier-real-googlebot-or-fa": {
      "type": "http",
      "url": "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/netcheck"
    }
  }
}

Tools

9

no_arguments_report_the_crawler_ip_ranges_this_host_mirrors

No arguments: every operator prefix list this host mirrors, and how stale each one is

TAKES NO ARGUMENTS. POST {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_report_the_crawler_ip_ranges_this_host_mirrors","arguments":{}}} to https://www.pathwren.workers.dev/mcp/netcheck — the answer is the state of every crawler-operator prefix list this host mirrors — how many IPv4 and IPv6 prefixes each source publishes and how many addresses that is, when each was last fetched and how many minutes ago that was, which crawlers each source covers, which prefixes more than one operator claims, and the verification method each operator documents for the crawlers that publish no list at all. There is nothing to fill in: the input schema is literally empty, `arguments: {}` and no `arguments` key at all both work, and the subject is a file this host already publishes, so the answer does not depend on you at all. No key, no account, no OAuth, no session to open first, read-only, and nothing for you to invent. Nothing is fetched to build it — no request leaves this edge, and none is made to you. The other zero-argument call on this server is verify_my_address, same empty arguments, which answers the address you are calling from, checked against every operator-published prefix list this host mirrors with your user-agent taken as the claim — the verdict, the prefix that matched, which operator published it and when it was last mirrored. whoami and example are here too and take nothing either. Every other tool on this server wants a file pasted in; this one wants nothing. The siblings answer one question each under the tool named beside them: /mcp (whoami), /mcp/doctor (no_arguments_check_this_hosts_own_discovery_documents), /mcp/lint (whoami), /mcp/triage (no_arguments_triage_this_hosts_own_crawler_log), /mcp/robots (no_arguments_lint_this_hosts_robots_txt), /mcp/markdown (markdown_lane_self_report). Example: the complete call, exactly as written, nothing to fill in — {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_

Takes no parameters.

verify_my_address

Does my address verify? (no arguments)

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"verify_my_address","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp/netcheck and the answer is the address you are calling from, checked against every operator-published prefix list this host mirrors with your user-agent taken as the claim — the verdict, the prefix that matched, which operator published it and when it was last mirrored. THIS IS THE CALL TO MAKE FIRST on this server: it is the only one whose subject is the request you just sent, so there is nothing to look up and no argument to invent. No key, no account, no OAuth, no session to open first, read-only, and nothing for you to invent. Every fact is computed from your own headers and from files this host already publishes; nothing is fetched to build it, and no URL of yours is touched. Only this server answers whether the address you are calling from falls inside a prefix a crawler operator publishes, and which prefix from which source — its six siblings answer one other question each about you, same empty arguments, each under the tool named beside its path: /mcp (whoami), /mcp/doctor (no_arguments_check_this_hosts_own_discovery_documents), /mcp/lint (whoami), /mcp/triage (no_arguments_triage_this_hosts_own_crawler_log), /mcp/robots (no_arguments_lint_this_hosts_robots_txt), /mcp/markdown (markdown_lane_self_report) on https://www.pathwren.workers.dev; whoami is answered by all seven and is never an error, so one identical body still walks the whole family if you prefer that. Example: the complete call, exactly as written, nothing to fill in — {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"verify_my_address","arguments":{}}} returns a verdict for the address you are calling from, the matching prefix with its source and mirror time, and the sentence that a miss is not evidence of a fake.

Takes no parameters.

whoami

Who is calling? (no arguments)

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp/netcheck and you are done: free, read-only, no API key, no account, no consent prompt, nothing for you to invent, and what comes back describes YOU, the caller — the user-agent you sent, the address you came from, the class our instrument books you as, and whether this host has seen you before. Takes no arguments. Safe to call. Deterministic. Touches no third party. Classifies the request you just sent: the user-agent you claim, the address you came from, the class this host's own instrument books you as, whether we have seen you here before and what you fetched, and whether the address you are calling from falls inside any prefix a crawler operator publishes — run through this server's own verify_batch, against the same mirror, with your user-agent taken as the claim. Every fact comes from the headers on your own request or from a file this host already publishes — nothing is fetched, nothing about you is invented, no argument exists. Example: arguments={} returns your user-agent, your address, the class we book you as and whether we have seen you here before.

Takes no parameters.

example

Run this server's worked example (no arguments)

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"example","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp/netcheck and you are done: free, read-only, no API key, no account, no consent prompt, nothing for you to invent, and what comes back is this server's own worked example — verify_batch, run for real on a canned input from this host's own data — so you see the exact shape of an answer before you send anything of yours. Takes no arguments. Safe to call. Deterministic. Touches no third party. Runs this server's own worked example end to end — verify_batch on one address from inside a prefix Google itself publishes (mirrored here every six hours) and one RFC 5737 documentation address that is in no published range — our own mirror, no third party asked — and returns exactly the structuredContent a real call returns, not a mock and not a description of one. Use it to see the shape of an answer before you decide what to send. The input is canned from this host's own data; no URL of yours is fetched and no third party is touched. Example: arguments={} runs verify_batch with {"addresses":[{"ip":"66.249.66.1","claim":"Googlebot"},{"ip":"203.0.113.9","claim":"GPTBot and returns its real answer.

Takes no parameters.

verify_batch

Verify many addresses at once

FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_report_the_crawler_ip_ranges_this_host_mirrors","arguments":{}}} — Check up to 500 addresses against every mirrored operator prefix list in one call. Returns, per address, the matching prefix, the source that published it and when it was mirrored, plus a rollup by operator. Accepts {ip, claim} pairs to test a claimed identity against the ranges, and says plainly that a miss is not evidence of a fake. Example: addresses=[{ip:'66.249.66.1', claim:'Googlebot'}, {ip:'203.0.113.9', claim:'GPTBot'}] — the first matches a published Google prefix, the second matches nothing. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/verify-crawler?ip=66.249.66.1&ua=Googlebot&s=client-dossiers

Parametersaddressesrequired

lookup_prefix

Prefix arithmetic against published ranges

FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_report_the_crawler_ip_ranges_this_host_mirrors","arguments":{}}} — Give a CIDR and get every published operator prefix that contains it, is contained by it, or partially overlaps it — the check for 'does my network collide with a crawler range' and for auditing an allowlist somebody handed you. Give an operator or source slug instead and get everything that source publishes. Example: cidr='66.249.66.0/24'.

Parameterscidroperator

export_ip_acl

Paste-ready allowlist or denylist

FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_report_the_crawler_ip_ranges_this_host_mirrors","arguments":{}}} — Turn a set of operators into a config you can paste: nginx geo, nginx allow/deny, Apache, HAProxy, a Cloudflare firewall expression, an ipset script, a Caddy matcher, or a plain CIDR list. Every export carries a provenance header naming each source URL and the mirror time, and reports the rule cost. Example: operators='all', format='cidr-list', action='allow'.

Parametersactionformatoperatorsip_versionvariable_name

verification_methods

Can this crawler be verified at all, and how?

FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_report_the_crawler_ip_ranges_this_host_mirrors","arguments":{}}} — Per crawler: whether the operator publishes IP ranges, documents reverse DNS, or publishes no verification method at all — with the exact source URL or the exact dig commands, and how fresh this mirror is. Call it before reading anything into an address that matched nothing. Reverse DNS is never run here; the command is handed back instead. Example: crawler='claudebot'. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/verification-methods?crawler=claudebot&s=client-dossiers

Parameterscrawler

range_stats

The state of the mirror

Takes no arguments. Safe to call. Deterministic. Touches no third party. Per source: prefix counts, total addresses and /24-equivalents, smallest and largest prefix, the publishing URL and when it was last fetched; plus prefixes published by more than one operator. This is the sizing and freshness information an allowlist decision needs. Example: {} — no arguments at all.

Takes no parameters.

Version history

1
  1. v1.4.0LatestSep 16, 2026

Score

83% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 daysRequired25 pts
  • Auth declaredRequired10 pts
  • Tools listedRequired15 pts
  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts

    Declare a GitHub repository with a README, or serve resource_documentation in the OAuth protected-resource metadata.

  • LicenseNot applicable

    A hosted service is used under its provider's terms, so no license is expected here. This check is left out of the percentage.

Other Workers integrations