searxNcrawl

io.github.DasDigitaleMomentumPublished by GitHub account DasDigitaleMomentumv0.33.1Updated Oct 29, 2026

Web search via SearXNG and/or You.com, plus clean Markdown crawling via Crawl4AI. Self-hosted.

VerifiedSTDIODesktop onlyWeb Search & ScrapingBrowser Automation

Overview

AI-generated overview

Self-hosted MCP server that gives an assistant web search through SearXNG and/or You.com plus clean Markdown crawling of pages and sites.

What you can do
  • Search the web across configured providers in parallel, merged and attributed per provider, with language, time range, safe search and paging filters
  • Crawl one or many URLs in Chromium and return Markdown or JSON with metadata and references
  • Crawl whole sites from a seed URL within depth, page and timeout limits, tuned for documentation
  • Crawl pages behind a login using a saved browser session state
  • Also usable as a command-line tool and Python API
When to use it
Useful when an assistant needs fresh web results and readable page content without paying a hosted search or crawl API, especially for documentation research, summarising articles, or collecting a site's pages. It fits users who already run or are willing to run their own SearXNG instance, or who have a You.com key.
Requirements
Runs locally over STDIO via the uvx package (uv required). Search needs at least one provider: a SearXNG instance with JSON output enabled, set through SEARXNG_URL, and/or a You.com key in YDC_API_KEY. Crawling works without either. The first crawl downloads Playwright's Chromium (about 550 MB). Optional basic auth uses SEARXNG_USERNAME and SEARXNG_PASSWORD; SEARXNG_EXTRA_HEADERS adds custom headers.
Before you install
You.com queries are sent to You.com in the US, so search terms leave your machine. SEARXNG_PASSWORD, SEARXNG_EXTRA_HEADERS and YDC_API_KEY are secrets; header values can be loaded from secret files. Crawling fetches arbitrary URLs and can carry a saved login session, so captured session state should be protected. Optional HTTP transport is unauthenticated unless placed behind an authenticated TLS proxy with Host/Origin checks.

Installation

In SourceWeft

  1. Open searxNcrawl in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Desktop only via STDIO. STDIO servers start a local process, so they need the SourceWeft desktop host.

Other MCP clients

Follow the launch instructions in the repository.

README

[searxNcrawl: search and crawl the web for your AI agents. Open-source MCP server for web search and clean Markdown crawling.]

[PyPI] [Python 3.10+] [CI] [MIT licence] [MCP server]

Website · Quick start · Docs · Developers · Changelog

[An MCP client starts uvx searxncrawl, calls search for crawl4ai quick start, then crawls the top result and receives clean Markdown]
A real MCP client session (fastmcp) against uvx searxncrawl 0.31.0 and a local SearXNG: search, then crawl the top hit into Markdown. Output is shortened for display. Reproduce it with docs/assets/demo/.

searxNcrawl is an open-source MCP server and CLI toolkit that gives AI agents web search and clean Markdown crawling, built on Crawl4AI. Search queries the configured providers, SearXNG and/or You.com, in parallel and merges the results. It runs on your own hardware, needs no API key with SearXNG and succeeds searxng-mcp.

Web search
Your own SearXNG instance and/or the You.com API, queried in parallel; results are merged and attributed to their providers, filtered by language and time range.		Page crawl
One or many URLs to Markdown or JSON, with repeated blocks removed and links optionally stripped.		Site crawl
Whole sites with depth and page limits, tuned for documentation.
Authenticated crawls
Capture a login session once with crawl-capture, then crawl behind the login.		STDIO and HTTP
Plug into local MCP clients, or run an HTTP endpoint with opt-in Host/Origin checks and configurable CORS.		CLI and Python API
crawl, search and crawl-capture on the command line, or import it in Python.
bash
SEARXNG_URL=http://your-searxng:8888 uvx searxncrawl

Quick start

Connect your MCP client directly to searxNcrawl over STDIO or its HTTP endpoint.

You need uv and, for the search tool, at least one search provider: a SearXNG instance with JSON output enabled and/or a You.com API key (YDC_API_KEY). No SearXNG yet? Start one with Docker. Crawling works without either.

Add it to your agent

One step for every agent on your machine

bash
uvx searxncrawl setup

setup detects Claude Code, Codex, Gemini CLI, Cursor, VS Code and Claude Desktop, asks for your SearXNG URL and, optionally, a You.com API key (hidden input, never printed), shows each config change and applies it only after you confirm. Every file it touches is backed up first, and setup --remove undoes it. More options are in MCP server.

Or per agent

[Add searxNcrawl to Cursor] [Install searxNcrawl in VS Code]

The buttons and commands below assume SearXNG at http://localhost:8888; change SEARXNG_URL if yours runs elsewhere, or add YDC_API_KEY for You.com.

Claude Code

bash
claude mcp add searxncrawl --env SEARXNG_URL=http://localhost:8888 -- uvx searxncrawl

Codex

bash
codex mcp add searxncrawl --env SEARXNG_URL=http://localhost:8888 -- uvx searxncrawl

VS Code

bash
code --add-mcp '{"name":"searxncrawl","command":"uvx","args":["searxncrawl"],"env":{"SEARXNG_URL":"http://localhost:8888"}}'

Cursor (~/.cursor/mcp.json), Claude Desktop and other clients that use the mcpServers format

json
{  "mcpServers": {    "searxncrawl": {      "command": "uvx",      "args": ["searxncrawl"],      "env": { "SEARXNG_URL": "http://localhost:8888" }    }  }}

The first crawl downloads Playwright's Chromium once (about 550 MB). For Docker, HTTP transport and other setups see Installation and MCP server.

  • Claude Desktop bundle: download searxncrawl-<version>.mcpb from GitHub Releases and open it in Claude Desktop.
  • Smithery users: use the direct installation above or the native bundle; see the Smithery transition.

Ask your agent

text
Search for the Crawl4AI quick start, crawl the official page and summarisehow to run a first crawl.

Your agent calls search, picks a result and calls crawl. It gets back clean Markdown instead of raw HTML.

Use the terminal

bash
uvx --from searxncrawl crawl https://modelcontextprotocol.io/docs/getting-started/introSEARXNG_URL=http://localhost:8888 uvx --from searxncrawl search "model context protocol" --max-results 5SEARXNG_URL=http://localhost:8888 uvx --from searxncrawl search "Python documentation" --max-results 3 --include-content

Choose how to connect

ModeSetupHosting
Native MCP over STDIO (default)uvx searxncrawl or uvx searxncrawl setupRuns alongside your MCP client
Native MCP over HTTPuvx searxncrawl --transport http or Docker ComposeYou operate the server; see HTTP configuration
Arcade gateway (optional)Separate Arcade adapterRequires your own Playwright-compatible worker host and authenticated connection to Arcade
  • Standard installations and releases do not require Arcade.
  • For shared HTTP access, use an authenticated TLS proxy. Enable Host/Origin checks with FASTMCP_HTTP_HOST_ORIGIN_PROTECTION=true before startup; allowlists alone do not enable checks or authenticate callers.

How it works

mermaid
flowchart LR    agent["AI agent<br/>Claude Code · Codex<br/>Cursor · VS Code"]    server["searxNcrawl<br/>MCP server"]    searxng["SearXNG (your instance)<br/>and/or You.com API"]    browser["Crawl4AI +<br/>Playwright Chromium"]    results["Merged results<br/>attributed per provider"]    markdown["Deduplicated<br/>Markdown or JSON"]
    agent -- "MCP · STDIO or HTTP" --> server    server -- "search" --> searxng --> results    server -- "crawl · crawl_site" --> browser --> markdown

Results return to the agent as the tool response. Search and crawling are also available as the search and crawl CLI commands, crawling also as a Python API.

searxNcrawl exposes three MCP tools:

ToolWhat it does
searchQueries the configured providers (SearXNG, You.com) in parallel and returns one merged, deduplicated list with the source of each result. Filters: language, time range, safe search, paging; categories and engines for SearXNG. With include_content, also returns the Markdown of each result page in the same call.
crawlRenders one or many URLs in Chromium and returns Markdown or JSON with metadata and references.
crawl_siteFollows links from a seed URL within depth, page and timeout limits, tuned for documentation sites.

Both crawl tools drop sections that repeat within a page's Markdown (dedup_mode="exact"), can strip links for a smaller context (remove_links), and accept a Playwright storage_state for pages behind a login. Full parameters are in the MCP tools reference.

Private SearXNG instances

  • SEARXNG_EXTRA_HEADERS adds headers to SearXNG requests in both CLI and MCP, including access-gateway credentials and a custom User-Agent.
  • Values such as @~/.config/searxncrawl/cf_secret load secrets from files; absolute and ~/ paths are supported.
  • Invalid header configuration disables SearXNG with a value-free warning; a configured You.com provider remains available.
  • Setup and JSON examples: SearXNG configuration.

Why searxNcrawl

Hosted search and crawl APIs are quick to start with, but every call goes through someone else's service and is billed per request or credit. searxNcrawl runs on your own machine or server.

searxNcrawlFirecrawlTavilyExa
Search and crawl run on your hardware✅ Python package or Docker; You.com queries go to You.com✅ Self-host stack (Docker Compose), or hosted API❌ Hosted API❌ Hosted API
API keyNot needed with SearXNG; You.com needs oneHosted API: needed. Self-hosted: not by defaultNeededOptional, anonymous use is rate-limited
Cost per callNone with SearXNG; You.com per its plansHosted: free monthly credits, then paid credits. Self-hosted: noneFree monthly credits, then paid creditsFree monthly credits, then paid per request
Search backendYour SearXNG (the engines you enable) and/or You.comHosted: Firecrawl's search. Self-hosted: SearXNG if configured, otherwise DuckDuckGoTavily's search APIExa's search index
LicenceMITAGPL-3.0MCP server MIT, API proprietaryMCP server MIT, API proprietary

Self-hosted Firecrawl can search through SearXNG too; it runs as a multi-service stack (API, Playwright service, Redis, RabbitMQ, PostgreSQL and more), while searxNcrawl is a single Python process with Chromium, plus your SearXNG instance if you use one.

Where the others are a better fit: if you do not want to run a search provider and a browser yourself, a hosted API is less work, and Exa's semantic search index or Firecrawl's extraction features go beyond what searxNcrawl does today. Comparison based on each project's public repository and documentation in October 2026; corrections are welcome. Firecrawl is a trademark of its owner; searxNcrawl is not affiliated with it.

FAQ

Do I need SearXNG?

Only for the search tool, and only if you do not use You.com (YDC_API_KEY). crawl and crawl_site work without any search provider. SearXNG serves HTML only by default, so JSON output must be enabled (search.formats in settings.yml). Public instances may not enable it and often rate-limit automated requests, so run your own. Installation has a Docker one-liner.

Does it handle JavaScript-heavy pages?

Yes. Pages are rendered in Playwright's Chromium through Crawl4AI before they are converted to Markdown, so client-side content is captured.

Can it crawl pages behind a login?

Yes. Capture a session once with crawl-capture, then pass the saved storage_state file to crawl or crawl_site. See Session capture.

Can I run it remotely or share it with a team?
  • Start with --transport http (port 8000 by default) or Docker Compose (port 9555).
  • Put shared access behind an authenticated TLS proxy and connect clients to its /mcp endpoint.
  • Enable header protection and configure the externally visible Host and Origin allowlists.
Why is the first crawl slow?

The first crawl downloads Playwright's Chromium (about 550 MB on disk). Later runs reuse it. Set PLAYWRIGHT_AUTO_INSTALL=false where browsers are provisioned separately.

I used searxng-mcp. What changed?

searxng-mcp is deprecated in favour of searxNcrawl. It only searched SearXNG and could pick a random public instance; searxNcrawl adds page and site crawling but needs your own SEARXNG_URL (or a You.com key), as random public-instance selection is not included.

Do I need Arcade?
  • No. Connect directly over STDIO or HTTP for the standard setup.
  • The optional adapter connects a separately hosted worker to Arcade's authentication and gateway.
  • Arcade Cloud's current runner cannot install Playwright; the worker needs a compatible host.

Documentation

GuideContents
Installationuvx, Docker Compose, pip, uv, search providers and environment variables
MCP serverSTDIO and HTTP transports, client configuration, Host and CORS guards
MCP toolsParameters for crawl, crawl_site and search
CLIcrawl, search and crawl-capture
Python APIcrawl_page, crawl_site and crawler configuration overrides
Output formatsMarkdown and JSON structure, CrawledDocument
Session captureAuthenticated crawling with a saved browser session
MCP Registryserver.json and the publish steps
Arcade adapter (optional)Shared authentication and routing through Arcade; separate deployment
ArchitectureModules and features for contributors

Contributing

Bug reports, feature ideas and pull requests are welcome. Read the contributing guide and open an issue first for larger changes. Coding agents find project conventions in AGENTS.md. Report vulnerabilities privately as described in the security policy.

License

MIT — © 2026 DDM – Das Digitale Momentum GmbH & Co KG

Maintained by DDM – Das Digitale Momentum GmbH & Co. KG, Much, Germany · www.searxncrawl.io · All our open-source projects

Source: README.md at commit d4d5d6c

Tools

0
SourceWeft has not started this package in its sandbox yet, so its tools are not known.

Version history

2
  1. v0.33.1LatestOct 29, 2026
  2. v0.32.0Oct 16, 2026

Score

50% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 days25 pts

    Keep the endpoint answering the MCP handshake; SourceWeft probes it every 30 days.

  • Auth declared10 pts

    Declare the credential headers in the registry entry, or keep the endpoint answering the probe so its auth can be classified.

  • Tools listed15 pts

    Answer tools/list without a credential, or ship a package that starts in a sandbox without one.

  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts