searxNcrawl

[searxNcrawl: search and crawl the web for your AI agents. Open-source MCP server for web search and clean Markdown crawling.]

[PyPI] [Python 3.10+] [CI] [MIT licence] [MCP server]

Website · Quick start · Docs · Developers · Changelog

[An MCP client starts uvx searxncrawl, calls search for crawl4ai quick start, then crawls the top result and receives clean Markdown]
_{A real MCP client session (fastmcp) against uvx searxncrawl 0.31.0 and a local SearXNG: search, then crawl the top hit into Markdown. Output is shortened for display. Reproduce it with docs/assets/demo/.}

searxNcrawl is an open-source MCP server and CLI toolkit that gives AI agents web search and clean Markdown crawling, built on Crawl4AI. Search queries the configured providers, SearXNG and/or You.com, in parallel and merges the results. It runs on your own hardware, needs no API key with SearXNG and succeeds searxng-mcp.

SEARXNG_URL=http://your-searxng:8888 uvx searxncrawl

Quick start

Connect your MCP client directly to searxNcrawl over STDIO or its HTTP endpoint.

You need uv and, for the search tool, at least one search provider: a SearXNG instance with JSON output enabled and/or a You.com API key (YDC_API_KEY). No SearXNG yet? Start one with Docker. Crawling works without either.

Add it to your agent

One step for every agent on your machine

uvx searxncrawl setup

setup detects Claude Code, Codex, Gemini CLI, Cursor, VS Code and Claude Desktop, asks for your SearXNG URL and, optionally, a You.com API key (hidden input, never printed), shows each config change and applies it only after you confirm. Every file it touches is backed up first, and setup --remove undoes it. More options are in MCP server.

Or per agent

[Add searxNcrawl to Cursor] [Install searxNcrawl in VS Code]

The buttons and commands below assume SearXNG at http://localhost:8888; change SEARXNG_URL if yours runs elsewhere, or add YDC_API_KEY for You.com.

Claude Code

claude mcp add searxncrawl --env SEARXNG_URL=http://localhost:8888 -- uvx searxncrawl

Codex

codex mcp add searxncrawl --env SEARXNG_URL=http://localhost:8888 -- uvx searxncrawl

VS Code

code --add-mcp '{"name":"searxncrawl","command":"uvx","args":["searxncrawl"],"env":{"SEARXNG_URL":"http://localhost:8888"}}'

Cursor (~/.cursor/mcp.json), Claude Desktop and other clients that use the mcpServers format

{  "mcpServers": {    "searxncrawl": {      "command": "uvx",      "args": ["searxncrawl"],      "env": { "SEARXNG_URL": "http://localhost:8888" }    }  }}

The first crawl downloads Playwright's Chromium once (about 550 MB). For Docker, HTTP transport and other setups see Installation and MCP server.

• Claude Desktop bundle: download searxncrawl-<version>.mcpb from GitHub Releases and open it in Claude Desktop.
• Smithery users: use the direct installation above or the native bundle; see the Smithery transition.

Ask your agent

Search for the Crawl4AI quick start, crawl the official page and summarisehow to run a first crawl.

Your agent calls search, picks a result and calls crawl. It gets back clean Markdown instead of raw HTML.

Use the terminal

uvx --from searxncrawl crawl https://modelcontextprotocol.io/docs/getting-started/introSEARXNG_URL=http://localhost:8888 uvx --from searxncrawl search "model context protocol" --max-results 5SEARXNG_URL=http://localhost:8888 uvx --from searxncrawl search "Python documentation" --max-results 3 --include-content

Choose how to connect

• Standard installations and releases do not require Arcade.
• For shared HTTP access, use an authenticated TLS proxy. Enable Host/Origin checks with FASTMCP_HTTP_HOST_ORIGIN_PROTECTION=true before startup; allowlists alone do not enable checks or authenticate callers.

How it works

flowchart LR    agent["AI agent<br/>Claude Code · Codex<br/>Cursor · VS Code"]    server["searxNcrawl<br/>MCP server"]    searxng["SearXNG (your instance)<br/>and/or You.com API"]    browser["Crawl4AI +<br/>Playwright Chromium"]    results["Merged results<br/>attributed per provider"]    markdown["Deduplicated<br/>Markdown or JSON"]
    agent -- "MCP · STDIO or HTTP" --> server    server -- "search" --> searxng --> results    server -- "crawl · crawl_site" --> browser --> markdown

Results return to the agent as the tool response. Search and crawling are also available as the search and crawl CLI commands, crawling also as a Python API.

searxNcrawl exposes three MCP tools:

Both crawl tools drop sections that repeat within a page's Markdown (dedup_mode="exact"), can strip links for a smaller context (remove_links), and accept a Playwright storage_state for pages behind a login. Full parameters are in the MCP tools reference.

Private SearXNG instances

• SEARXNG_EXTRA_HEADERS adds headers to SearXNG requests in both CLI and MCP, including access-gateway credentials and a custom User-Agent.
• Values such as @~/.config/searxncrawl/cf_secret load secrets from files; absolute and ~/ paths are supported.
• Invalid header configuration disables SearXNG with a value-free warning; a configured You.com provider remains available.
• Setup and JSON examples: SearXNG configuration.

Why searxNcrawl

Hosted search and crawl APIs are quick to start with, but every call goes through someone else's service and is billed per request or credit. searxNcrawl runs on your own machine or server.

Self-hosted Firecrawl can search through SearXNG too; it runs as a multi-service stack (API, Playwright service, Redis, RabbitMQ, PostgreSQL and more), while searxNcrawl is a single Python process with Chromium, plus your SearXNG instance if you use one.

Where the others are a better fit: if you do not want to run a search provider and a browser yourself, a hosted API is less work, and Exa's semantic search index or Firecrawl's extraction features go beyond what searxNcrawl does today. Comparison based on each project's public repository and documentation in October 2026; corrections are welcome. Firecrawl is a trademark of its owner; searxNcrawl is not affiliated with it.

FAQ

Documentation

Contributing

Bug reports, feature ideas and pull requests are welcome. Read the contributing guide and open an issue first for larger changes. Coding agents find project conventions in AGENTS.md. Report vulnerabilities privately as described in the security policy.

License

MIT — © 2026 DDM – Das Digitale Momentum GmbH & Co KG

_{Maintained by DDM – Das Digitale Momentum GmbH & Co. KG, Much, Germany · www.searxncrawl.io · All our open-source projects}