Webcrawl Anything

io.github.brainfish-aiPublished by GitHub account brainfish-aiv1.4.3Updated Nov 7, 2026

Scrape, map, crawl, search and monitor the web. Runs locally with no API key, cites every source.

VerifiedStreamable HTTPWeb executableWeb Search & ScrapingKnowledge & Memory

Overview

AI-generated overview

Webcrawl Anything lets an assistant search the web, read pages, crawl sites, extract structured data and parse documents, locally or via a hosted cloud.

What you can do
  • Search the web and structured verticals such as shopping, flights, hotels, real estate, business, security, academic, tech and news, and list available categories and filters
  • Fetch a URL as clean Markdown with cited evidence, and rerank candidate documents by relevance to a query
  • Discover a domain's URLs from sitemaps, robots.txt and the link graph, and crawl up to 25 pages from a starting URL
  • Extract structured JSON from a page using a prompt or JSON Schema
  • Parse PDFs with OCR, Word, Excel and PowerPoint files, YouTube videos, and audio or video into Markdown
  • Monitor a page, URL list or site on a schedule with trigger conditions and email or webhook alerts (cloud only)
When to use it
Useful when an assistant needs live web information, clean readable page content, site-wide URL discovery, or structured data pulled from pages and documents. It suits research, competitive or price lookups, and recurring page monitoring without building a separate scraping stack.
Requirements
Runs locally through the npm package @webcrawl/mcp via npx, with no API key or account needed. The hosted option needs either a WEBCRAWL_API_KEY for the cloud service or a browser sign-in to the remote endpoint. WEBCRAWL_MODE selects local, cloud or auto; WEBCRAWL_MCP_SCHEMA_PROFILE=compact reduces tool schema size.
Before you install
The hosted cloud requires an account and an API key (WEBCRAWL_API_KEY) or OAuth sign-in, and cloud-only monitoring can send email or webhook alerts. Crawling and monitoring target third-party sites, so respect their terms, robots directives and rate limits. Page content and parsed documents are sent to the chosen backend, local or hosted.

Installation

In SourceWeft

  1. Open Webcrawl Anything in the dashboard and add it to a workspace.
  2. Connect the required credentials. They are stored privately and never shown on public pages.
  3. Enable the server for the chats that should use its tools.

Web executable via Streamable HTTP. Remote servers run from the web runtime once configured in a workspace.

Use with an agent

Paste this prompt into Claude Code, Codex, Cursor or another agent — or into SourceWeft chat. The agent reads this server's install guide, shows you its publisher, endpoint and sign-in, and adds it to your client once you agree. Keys never go into a command.

Read https://sourceweft.com/mcp/io.github.brainfish-ai%2Fwebcrawl/install.md and connect the MCP server it describes to my client. Before changing any config, show me its publisher, endpoint and sign-in method and wait for my OK. Never put an API key in a command or in a file I would commit; ask me where it should come from.

Read the install guide the agent follows

Other MCP clients

Add this to your client's mcpServers config. The client completes OAuth sign-in on first connect.

{
  "mcpServers": {
    "webcrawl-anything": {
      "type": "http",
      "url": "https://webcrawl.app/mcp"
    }
  }
}

README

@webcrawl/mcp

Model Context Protocol (MCP) server for Webcrawl Anything.

Gives Claude Code, Cursor, Codex, Windsurf, and Gemini CLI live access to web search, structured verticals (shopping, flights, hotels, real-estate, local business, CVEs), page reading for JavaScript-heavy sites, and PDF/Office parsing. It runs on your machine with no API key and no account.

One-Command Agent Setup

Register @webcrawl/mcp and install the webcrawl-anything Agent Skill (SKILL.md) automatically:

bash
npx webcrawl setup

Manual MCP Configuration

Add to .cursor/mcp.json, ~/.claude.json, or your MCP client config:

json
{  "mcpServers": {    "webcrawl": {      "command": "npx",      "args": ["-y", "@webcrawl/mcp@latest"],      "env": {        "WEBCRAWL_MCP_SCHEMA_PROFILE": "compact"      }    }  }}

Optional: webcrawl.app cloud

Local mode is the default and needs nothing else. The packaged server loads the local engine from dist/core; if that import fails it exits with an error instead of silently calling webcrawl.app. To run the same tools on the hosted service, either:

  • add "WEBCRAWL_API_KEY": "wc_live_..." to env above (get a key at webcrawl.app), or
  • connect your client to the hosted endpoint https://webcrawl.app/mcp and sign in in the browser. No local install is needed.

WEBCRAWL_MODE can be local, cloud or auto (default: local unless a key is set).

Compact Schema Profile

Set WEBCRAWL_MCP_SCHEMA_PROFILE=compact (default when installed via npx webcrawl setup) to minimize tool schema token usage in context-constrained agents while keeping full server-side validation. Set WEBCRAWL_MCP_SCHEMA_PROFILE=full for verbose parameter descriptions.

Tools Provided

  • web_search — Search the web or structured verticals (shopping, flights, hotels, real-estate, business, security, academic, tech, news).
  • web_read — Fetch a URL and return clean Markdown with target evidence.
  • web_parse / web_parse_status — Parse PDFs (native + OCR), Word/Excel/PowerPoint files, YouTube videos, and audio/video into Markdown.
  • web_map — Discover all URLs on a domain (sitemap.xml, robots.txt, and link graph).
  • web_crawl — Crawl up to 25 pages starting from a URL.
  • web_extract — Extract structured JSON from a URL using a prompt or JSON Schema.
  • web_rerank — Rerank candidate documents by relevance to a query.
  • web_monitor — Watch a page, URL list or site on a schedule with trigger conditions and email/webhook alerts (cloud only).
  • search_categories — List all supported search categories, subcategories, and filters.

License

MIT

Source: packages/mcp/README.md at commit a1c1632

Tools

0
This server asks you to sign in, so SourceWeft could not read its tools without an account and has not indexed them. What a connected account sees can depend on that account and its permissions.

Version history

4
  1. v1.4.3LatestNov 7, 2026
  2. v1.4.2Nov 4, 2026
  3. v1.4.1Nov 3, 2026
  4. v1.3.9Nov 3, 2026

Score

85% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 daysRequired25 pts
  • Auth declaredRequired10 pts
  • Tools listed15 pts

    Answer tools/list without a credential, or ship a package that starts in a sandbox without one.

  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts