Deepcrawl

com.soflutionPublished by soflution.comv1.1.1Updated Oct 29, 2026

Clone websites, scrape pages to markdown and crawl whole sites. A free Firecrawl alternative.

VerifiedSTDIODesktop onlyWeb Search & ScrapingFiles & Storage

Overview

AI-generated overview

Local MCP server that lets an assistant scrape pages to markdown, clone pages with assets, crawl sites and map URLs, without API keys.

What you can do
  • Scrape a single page into clean markdown with title, description, links, images and metadata, stripping navigation, ads and tracking
  • Clone a page with all its assets into a local folder and rewrite URLs to relative paths, optionally following links to a set depth
  • Crawl a site following internal links and return each page as markdown, up to a page limit
  • Map a site's URLs from sitemap.xml and homepage links to see its scope before crawling
When to use it
Useful when you want an assistant to pull web content into markdown for content analysis, SEO audits or RAG pipelines, or to save a working local copy of a page, without signing up for a paid scraping service.
Requirements
Runs locally over stdio; the README shows an npx command, so Node.js and npm are needed. No API keys or environment variables are declared. JS-heavy sites need Playwright with Chromium installed globally to enable rendering. Network access to the target sites and to registry.npmjs.org for the update check.
Before you install
The server auto-updates itself: a supervisor downloads the latest package from npm at start-up and every 6 hours into a local folder and switches over; disable with NO_UPDATE_NOTIFIER=1 or SOFLUTION_AUTO_UPDATE=0. It writes cloned files to a local output folder. It uses user-agent rotation, browser headers and delays to avoid basic bot detection, so respect site terms and robots rules. Crawling sends requests to third-party sites.

Installation

In SourceWeft

  1. Open Deepcrawl in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Desktop only via STDIO. STDIO servers start a local process, so they need the SourceWeft desktop host.

Other MCP clients

Follow the launch instructions in the repository.

README

[deepcrawl]

[npm version] [4 tools] [free] [no API keys]

[Install in Cursor] [Install via Web]

Clone, scrape and crawl any website. Free Firecrawl alternative. No API keys, no rate limits, no subscription.

Quick Start

Cursor (one-click): Click the "Install in Cursor" button above.

Manual MCP config:

json
{  "mcpServers": {    "deepcrawl-mcp": {      "command": "npx",      "args": ["-y", "deepcrawl-mcp@latest"]    }  }}

Automatic updates

Every install below keeps itself up to date, without restarting Claude or Cursor. The MCP client talks to a small supervisor (bin/supervise.cjs) that stays connected while the server runs behind it. At start-up and every 6 hours it asks npm for the latest deepcrawl-mcp, installs it in the background in ~/.soflution/mcp/deepcrawl-mcp/, and switches over as soon as no request is in flight (the client is told the tool list may have changed). A version that fails to start is rolled back at once and never picked again on that machine. npm checks the integrity of every download, and releases are published from GitHub Actions with provenance.

The only network call is that version check to registry.npmjs.org; nothing about your machine or your use is sent. Turn it off with NO_UPDATE_NOTIFIER=1 (or SOFLUTION_AUTO_UPDATE=0). Started directly with node dist/..., deepcrawl-mcp does not update itself and only mentions a newer version in its first tool result.

CLI:

bash
npx deepcrawl-mcp@latest

Tools

deepcrawl_scrape

Scrape a single page and return clean markdown. Extracts title, description, links, images, and metadata. Strips navigation, footer, ads, and tracking.

"Scrape https://example.com and give me the main content"
ParameterDefaultDescription
urlrequiredPage URL to scrape
mainContentOnlytrueExtract only main content (skip nav/footer)
includeLinkstrueInclude discovered links
includeImagestrueInclude image URLs

deepcrawl_clone

Clone a full page with all assets: HTML, CSS, JS, images, fonts, favicons. Downloads everything into a local folder, rewrites URLs to relative paths. Open index.html in a browser and it works.

"Clone https://competitor.com into a local folder"
ParameterDefaultDescription
urlrequiredPage URL to clone
outputDir~/deepcrawl-clones/<domain>Output folder
depth0Link depth: 0 = single page, 1+ = follow links

deepcrawl_crawl

Crawl an entire site following internal links. Returns every page as clean markdown. Great for content analysis, SEO audit, or feeding a RAG pipeline.

"Crawl https://docs.example.com and return all pages as markdown"
ParameterDefaultDescription
urlrequiredStarting URL
maxPages20Max pages to crawl (max: 100)
includeImagesfalseInclude image URLs per page

deepcrawl_map

Discover all URLs from a site via sitemap.xml parsing and homepage link crawling. Run this before a crawl to see the site's scope.

"Map all pages on https://example.com"
ParameterDefaultDescription
urlrequiredSite URL
maxUrls200Max URLs to discover

vs Firecrawl

deepcrawlFirecrawl
PriceFree$19+/mo
API keyNoneRequired
Rate limitsNoneYes
Scrape to markdownYesYes
Full site crawlYesYes
Site mapYesYes
Clone with assetsYesNo
JS renderingYes (via Playwright)Yes
Anti-bot bypassPartial (UA rotation, headers, delays)Yes

deepcrawl handles static sites out of the box. For JS-heavy SPAs (React, Next.js, SvelteKit), install Playwright for full rendering:

bash
npm install -g playwrightnpx playwright install chromium

Once installed, deepcrawl auto-detects Playwright and enables JS rendering. Use jsRender: true on any tool to activate it. All tools also include UA rotation, realistic browser headers, and random delays to avoid basic bot detection.

Also by Soflution

  • brandcheck - Check brand name availability across 27 platforms
  • depsonar - Dependency audit, security scan, license check

License

MIT

Source: README.md at commit 647fc33

Tools

0
SourceWeft has not started this package in its sandbox yet, so its tools are not known.

Version history

2
  1. v1.1.1LatestOct 29, 2026
  2. v1.1.0Oct 26, 2026

Score

50% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 days25 pts

    Keep the endpoint answering the MCP handshake; SourceWeft probes it every 30 days.

  • Auth declared10 pts

    Declare the credential headers in the registry entry, or keep the endpoint answering the probe so its auth can be classified.

  • Tools listed15 pts

    Answer tools/list without a credential, or ship a package that starts in a sandbox without one.

  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts