Crawlberg

[Xberg]

Crawlberg

Turn any website into clean, structured data. Point Crawlberg at a URL and get back Markdown, metadata, and links — from a single page or a whole site — in the language you already use.

What and Why?

You need data that lives on the web, and raw HTML is not it. Crawlberg does the crawling, scraping, and cleanup end-to-end: it fetches pages, follows links, converts each one to Markdown, and hands you structured metadata (titles, links, images, social-card and JSON-LD data) — so you skip the parsing and go straight to the content.

It runs from a single Rust core with identical results across 14 language bindings, and it handles the awkward parts for you: JavaScript-heavy pages fall back to a real headless browser, bot filters are detected and worked around, robots and sitemaps are respected, requests are throttled per domain, and requests to private or internal addresses are refused by default. Drive it from your code, an AI agent, a REST service, or the CLI.

Every part of the pipeline is a trait you can swap — the crawl frontier, rate limiter, storage, event stream, and content filters — so you can plug in your own behavior. Managed extras like proxy pools, tuned bot-evasion, authenticated sessions, scheduling, and billing live in xberg-enterprise.

Features

Supported Platforms

Precompiled binaries for glibc Linux (x86_64/aarch64), macOS, and Windows across every binding. On musl Linux (Alpine) coverage is deliberately partial — the CLI, Docker image, Rust, Ruby, Java, and Elixir work there; the Node, Python, Go, PHP, and C# bindings do not ship musl artifacts. See the platform support matrix for the full table and the Alpine workarounds.

⭐ Star this repo to show your support — it helps others discover Crawlberg.

Quick Start

Language Packages

pip install crawlberg

pnpm add @xberg-io/crawlberg

cargo add crawlberg

go get github.com/xberg-io/crawlberg/packages/go

dotnet add package XbergIo.Crawlberg

gem install crawlberg

composer require xberg-io/crawlberg

dart pub add crawlberg

pnpm add @xberg-io/crawlberg-wasm

cargo install crawlberg-cli

# Or install the prebuilt binary via cargo-binstall:cargo binstall crawlberg-cli

brew install xberg-io/tap/crawlberg

scoop bucket add xberg https://github.com/xberg-io/scoop-bucketscoop install crawlberg

AI Coding Assistants

Install the Crawlberg plugin from xberg-io/crawlberg. It ships the Crawlberg agent skills (site crawling, HTML→Markdown scraping, headless-Chrome fallback) plus the crawlberg MCP server, — select your runtime below.

The bundle targets Claude Code, Codex, Cursor, Gemini CLI, Kimi, OpenCode, Factory Droid, Hermes, Pi, Agent Plugins and GitHub Copilot CLI. Read the plugin guide for runtime resources, CLI prerequisites and configuration. MCP-capable adapters expose the product's tools; Hermes uses a local stdio bridge and explicitly selected skills. Pi loads skills and prompt templates and requires the product CLI for executable operations.

Install only the public Agent Skills, without a plugin or MCP registration, using the skills CLI:

npx skills add xberg-io/crawlberg --listnpx skills add xberg-io/crawlberg

Choose the skills and target agents interactively. Installing instructions does not install the product CLI or configure provider credentials.

/plugin marketplace add xberg-io/crawlberg/plugin install crawlberg@crawlberg

/plugins add https://github.com/xberg-io/crawlberg

gemini extensions install ./plugin

droid plugin marketplace add https://github.com/xberg-io/crawlbergdroid plugin install crawlberg@crawlberg

copilot plugin marketplace add https://github.com/xberg-io/crawlbergcopilot plugin install crawlberg@crawlberg

pi install ./plugin

pi install npm:@xberg-io/opencode-crawlberg

{  "$schema": "https://opencode.ai/config.json",  "plugin": ["@xberg-io/opencode-crawlberg"]}

Documentation

Full guides, per-language API references, the substrate/operational model, antibot strategy, and observability live at docs.crawlberg.xberg.io.

Contributing

Contributions are welcome! See our Contributing Guide.

Part of Xberg.io

• Xberg — the open-source content-intelligence engine: text, tables, and metadata from 101 formats (115 file extensions), with OCR, transcription, and code intelligence. MIT.
• Xberg Pro — a complete self-hosted content-intelligence backend in a single container. Commercial.
• Xberg Enterprise — the distributed, governed content-intelligence platform, scaled on Kubernetes with team governance and support. Commercial.
• crawlberg — web crawling and scraping with HTML→Markdown and headless-Chrome fallback.
• html-to-markdown — fast, lossless HTML→Markdown engine.
• liter-llm — universal LLM API client with native bindings for 14 languages and 165 providers.
• tree-sitter-language-pack — tree-sitter grammars and code-intelligence primitives.
• alef — the polyglot binding generator that produces every per-language binding across the 5 polyglot repos.

License

MIT License

Links

• Documentation
• GitHub Repository
• Issue Tracker
• Changelog
• Discord — community, roadmap, announcements.