Deepcrawl

io.github.SoflutionltdPublished by GitHub account Soflutionltdv1.1.0Updated Oct 26, 2026

Clone websites, scrape pages to markdown and crawl whole sites. A free Firecrawl alternative.

VerifiedSTDIODesktop onlyWeb Search & ScrapingFiles & Storage

Overview

AI-generated overview

Deepcrawl lets an assistant scrape pages to markdown, clone pages with assets, crawl whole sites and map URLs, without API keys.

What you can do
  • Scrape a single page into clean markdown with title, description, links, images and metadata, stripping navigation, ads and tracking
  • Clone a page with its HTML, CSS, JS, images and fonts into a local folder with rewritten relative URLs, optionally following links to a set depth
  • Crawl a site following internal links and return each page as markdown, up to a page limit
  • Map a site's URLs from sitemap.xml and homepage links to see its scope before crawling
  • Optionally render JavaScript-heavy pages through Playwright and apply user-agent rotation, browser headers and random delays
When to use it
Useful when you want an assistant to pull web content into markdown for analysis, SEO review or a RAG pipeline, or to keep a local copy of a page with all its assets, without signing up for a paid scraping service.
Requirements
Runs as a local process, typically started with npx (Node.js) or installed as an MCP bundle in a desktop client. No API keys or accounts. Network access to the target sites and to registry.npmjs.org for the update check. Full rendering of JavaScript-heavy sites needs Playwright with Chromium installed separately.
Before you install
The server updates itself in the background: at start-up and every 6 hours it queries npm and installs the latest version under a local folder, switching over between requests; disable this with NO_UPDATE_NOTIFIER=1 or SOFLUTION_AUTO_UPDATE=0. Cloning and crawling write files to a local output folder and fetch many pages from third-party sites, so respect site terms, robots rules and copyright. Anti-bot measures are only partial.

Installation

In SourceWeft

  1. Open Deepcrawl in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Desktop only via STDIO. STDIO servers start a local process, so they need the SourceWeft desktop host.

Other MCP clients

Follow the launch instructions in the repository.

README

[deepcrawl]

[npm version] [4 tools] [free] [no API keys]

[Install in Cursor] [Install via Web]

Clone, scrape and crawl any website. Free Firecrawl alternative. No API keys, no rate limits, no subscription.

Quick Start

Cursor (one-click): Click the "Install in Cursor" button above.

Manual MCP config:

json
{  "mcpServers": {    "deepcrawl-mcp": {      "command": "npx",      "args": ["-y", "deepcrawl-mcp@latest"]    }  }}

Automatic updates

Every install below keeps itself up to date, without restarting Claude or Cursor. The MCP client talks to a small supervisor (bin/supervise.cjs) that stays connected while the server runs behind it. At start-up and every 6 hours it asks npm for the latest deepcrawl-mcp, installs it in the background in ~/.soflution/mcp/deepcrawl-mcp/, and switches over as soon as no request is in flight (the client is told the tool list may have changed). A version that fails to start is rolled back at once and never picked again on that machine. npm checks the integrity of every download, and releases are published from GitHub Actions with provenance.

The only network call is that version check to registry.npmjs.org; nothing about your machine or your use is sent. Turn it off with NO_UPDATE_NOTIFIER=1 (or SOFLUTION_AUTO_UPDATE=0). Started directly with node dist/..., deepcrawl-mcp does not update itself and only mentions a newer version in its first tool result.

CLI:

bash
npx deepcrawl-mcp@latest

Tools

deepcrawl_scrape

Scrape a single page and return clean markdown. Extracts title, description, links, images, and metadata. Strips navigation, footer, ads, and tracking.

"Scrape https://example.com and give me the main content"
ParameterDefaultDescription
urlrequiredPage URL to scrape
mainContentOnlytrueExtract only main content (skip nav/footer)
includeLinkstrueInclude discovered links
includeImagestrueInclude image URLs

deepcrawl_clone

Clone a full page with all assets: HTML, CSS, JS, images, fonts, favicons. Downloads everything into a local folder, rewrites URLs to relative paths. Open index.html in a browser and it works.

"Clone https://competitor.com into a local folder"
ParameterDefaultDescription
urlrequiredPage URL to clone
outputDir~/deepcrawl-clones/<domain>Output folder
depth0Link depth: 0 = single page, 1+ = follow links

deepcrawl_crawl

Crawl an entire site following internal links. Returns every page as clean markdown. Great for content analysis, SEO audit, or feeding a RAG pipeline.

"Crawl https://docs.example.com and return all pages as markdown"
ParameterDefaultDescription
urlrequiredStarting URL
maxPages20Max pages to crawl (max: 100)
includeImagesfalseInclude image URLs per page

deepcrawl_map

Discover all URLs from a site via sitemap.xml parsing and homepage link crawling. Run this before a crawl to see the site's scope.

"Map all pages on https://example.com"
ParameterDefaultDescription
urlrequiredSite URL
maxUrls200Max URLs to discover

vs Firecrawl

deepcrawlFirecrawl
PriceFree$19+/mo
API keyNoneRequired
Rate limitsNoneYes
Scrape to markdownYesYes
Full site crawlYesYes
Site mapYesYes
Clone with assetsYesNo
JS renderingYes (via Playwright)Yes
Anti-bot bypassPartial (UA rotation, headers, delays)Yes

deepcrawl handles static sites out of the box. For JS-heavy SPAs (React, Next.js, SvelteKit), install Playwright for full rendering:

bash
npm install -g playwrightnpx playwright install chromium

Once installed, deepcrawl auto-detects Playwright and enables JS rendering. Use jsRender: true on any tool to activate it. All tools also include UA rotation, realistic browser headers, and random delays to avoid basic bot detection.

Also by Soflution

  • brandcheck - Check brand name availability across 27 platforms
  • depsonar - Dependency audit, security scan, license check

License

MIT

Source: README.md at commit 647fc33

Tools

0
SourceWeft started this package once in an isolated sandbox, without credentials, to list its tools. That failed: the package type cannot run in the sandbox.

Version history

1
  1. v1.1.0LatestOct 26, 2026

Score

50% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 days25 pts

    Keep the endpoint answering the MCP handshake; SourceWeft probes it every 30 days.

  • Auth declared10 pts

    Declare the credential headers in the registry entry, or keep the endpoint answering the probe so its auth can be classified.

  • Tools listed15 pts

    Answer tools/list without a credential, or ship a package that starts in a sandbox without one.

  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts