Rat Radar

tech.ratlabsPublished by ratlabs.techv1.4.0Updated Oct 30, 2026

Ask before you scrape: is a site open, may I fetch this URL, and how do I extract a page. No key.

VerifiedStreamable HTTPWeb executableWeb Search & ScrapingData & Analytics

Overview

AI-generated overview

Rat Radar advises an assistant on whether a public site can be scraped, whether a URL may be fetched, and how to extract data from a page.

What you can do
  • Check a site's scrapability: verdict, hostility score, bot protections, robots.txt and AI-crawler policy, machine-readable surfaces
  • Check whether robots.txt allows fetching a specific URL for a given user agent, with the deciding rule and crawl delay
  • Plan extraction from one public page: ranked sources such as embedded JSON, JSON-LD, repeated records with CSS selectors, tables and feeds, with proof rows
  • Get a workflow playbook for a domain and goal, and draw a page's anatomy as an Excalidraw diagram
  • Search and list the covered sites by name, host, category, country or verdict, and get headline dataset statistics
When to use it
Useful when an assistant is about to collect data from public websites and needs a quick read on whether a site permits it, which extraction route is most reliable, or what to build on top of a site. Also handy for surveying which sites are open or block AI crawlers.
Requirements
Remote streamable HTTP endpoint; no key, environment variables or headers declared. Network access to the endpoint and to the public pages it inspects.
Before you install
Verdicts come from a weekly heuristic probe and are not legal advice; robots.txt and site terms still apply. xray_page and can_fetch fetch live public pages, and xray_diagram returns a download link for a generated file.

Installation

In SourceWeft

  1. Open Rat Radar in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Web executable via Streamable HTTP. Remote servers run from the web runtime once configured in a workspace.

Use with an agent

Paste this prompt into Claude Code, Codex, Cursor or another agent — or into SourceWeft chat. The agent reads this server's install guide, shows you its publisher, endpoint and sign-in, and adds it to your client once you agree. Keys never go into a command.

Read https://sourceweft.com/mcp/tech.ratlabs%2Frat-radar/install.md and connect the MCP server it describes to my client. Before changing any config, show me its publisher, endpoint and sign-in method and wait for my OK. Never put an API key in a command or in a file I would commit; ask me where it should come from.

Read the install guide the agent follows

Other MCP clients

Add this to your client's mcpServers config.

{
  "mcpServers": {
    "rat-radar": {
      "type": "http",
      "url": "https://rat-radar.github-a18.workers.dev/mcp"
    }
  }
}

Tools

8

check_site

Can I scrape this site, and how? Returns a verdict (open, guarded, hostile, blocked), a hostility score, a recommended approach with steps and risks, bot protections, robots.txt and AI-crawler policy, machine-readable surfaces (sitemap, llms.txt, JSON-LD), and eu_ai_training: the signals the EDPB draft Guidelines 03/2026 on web scraping for generative AI count against collecting personal data (robots.txt, ai.txt, CAPTCHAs, login walls), as none, some or strong. Signals, not legal advice. Uses a weekly-probed dataset of several hundred sites; for any other domain it probes live from a datacenter IP.

Parametersdomainrequired

get_playbook

What should I build with this site, and how? Give a domain and, optionally, a goal in plain words (for example "price alerts", "sales leads", "new jobs", "change log", "knowledge base"). Returns the best-matching workflow for that site: what it does, its cadence, a trigger, collect, shape, store, act pipeline, the fields to keep, and step-by-step build recipes for n8n, Make, Zapier and Apify (or just one with platform). Workflows marked found_on_this_site come from what the weekly probe saw on the site itself. Also returns the other workflows, the golden data, and the guardrails.

Parametersgoaldomainrequiredplatform

list_sites

List the sites in the dataset, optionally by category or verdict. Use it to find easy targets or to see what is covered.

Parameterslimitverdictcategory

xray_page

How do I extract the data from THIS page? Fetches one public page and returns a ranked extraction plan: embedded JSON, JSON-LD, repeated records with CSS selectors and field names, tables, feeds, hidden API calls. Each source has a reliability, proof rows pulled from the real page, and the plan notes pagination and blockers. Set include_code for a starter Python script. Works on the raw HTML only, so JavaScript-built pages report a blocker instead of data. Use check_site first to learn whether the site allows scraping at all.

Parametersurlrequiredinclude_code

xray_diagram

Draw a page's anatomy as an Excalidraw diagram: its regions, the ranked ways to get its data, proof rows, the best route and what the page is made of. Returns a download link for the .excalidraw file (give it to the user; it opens at excalidraw.com or in Obsidian) and a file_name. With include_scene, also the scene itself (a few hundred KB).

Parametersurlrequiredinclude_scene

search_sites

Find sites in the dataset by a word: a name, host, category (jobs, shop, news, gov), country or verdict. Use it to discover what is covered before calling check_site.

Parameterslimitqueryrequired

can_fetch

Does robots.txt allow fetching this exact URL? Fetches the live robots.txt and applies the longest-match rule for a user agent (default: * plus GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot). Returns allowed or not, the rule that decided it, and any crawl delay. A courtesy check, not a legal opinion.

Parametersurlrequireduser_agent

summary

Headline numbers across the dataset: verdict mix, how many sites block AI crawlers (and which), protection vendors, by category and country.

Takes no parameters.

Version history

1
  1. v1.4.0LatestOct 30, 2026

Score

83% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 daysRequired25 pts
  • Auth declaredRequired10 pts
  • Tools listedRequired15 pts
  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts

    Declare a GitHub repository with a README, or serve resource_documentation in the OAuth protected-resource metadata.

  • LicenseNot applicable

    A hosted service is used under its provider's terms, so no license is expected here. This check is left out of the percentage.