Commoncrawl Mcp

io.github.mrfentmenPublished by GitHub account mrfentmenv1.0.1Updated Nov 5, 2026

Use this MCP server to Common Crawl index discovery for historical web captures. Tools include...

VerifiedSTDIODesktop onlyWeb Search & ScrapingKnowledge & Memory

Overview

AI-generated overview

Searches the Common Crawl index to find historical web captures and their metadata, without downloading page contents.

What you can do
  • List recent Common Crawl collections and return the newest collection id
  • Search URL or wildcard patterns for capture index metadata such as timestamp, status, digest, filename, offset and length
When to use it
Useful for provenance research, archival discovery and broken-link investigations, or for locating the capture metadata needed before fetching an archived page from Common Crawl.
Requirements
Runs as a local Node.js process installed from the npm package commoncrawl-mcp; no accounts, API keys or environment variables are declared. Network access to the Common Crawl index is needed.
Before you install
It only searches the index: it does not download page contents and does not guarantee that a capture is complete or legally reusable, so reuse rights must be checked separately. Requests and result pages are bounded.

Installation

In SourceWeft

  1. Open Commoncrawl Mcp in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Desktop only via STDIO. STDIO servers start a local process, so they need the SourceWeft desktop host.

Other MCP clients

Follow the launch instructions in the repository.

README

commoncrawl-mcp

Discover historical web captures through the Common Crawl index. This server is for provenance research, archival discovery, broken-link investigations, and finding the capture metadata needed for a later archive fetch.

Tools

  • list_indexes: list recent crawl collections.
  • latest_index: return the newest collection id.
  • search_captures: search URL or wildcard patterns and return capture index metadata such as timestamp, status, digest, filename, offset, and length.

This server searches the index only. It does not download page contents, bypass access controls, or guarantee that a capture is complete or legally reusable. Requests and result pages are bounded.

Run

bash
npm installnpm run buildnode dist/index.js

Source: README.md at commit 1259797

Tools

0
SourceWeft has not started this package in its sandbox yet, so its tools are not known.

Version history

1
  1. v1.0.1LatestSep 22, 2026

Score

40% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 days25 pts

    Keep the endpoint answering the MCP handshake; SourceWeft probes it every 30 days.

  • Auth declared10 pts

    Declare the credential headers in the registry entry, or keep the endpoint answering the probe so its auth can be classified.

  • Tools listed15 pts

    Answer tools/list without a credential, or ship a package that starts in a sandbox without one.

  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts

    Declare a license in the repository or the registry entry.