Crawlport

io.github.CloudCorpRecords由 GitHub 账号 CloudCorpRecords 发布v0.1.0更新于 Nov 3, 2026

Firecrawl web data: scrape, map, search, crawl, extract, research, and llms.txt.

已验证STDIO仅桌面网页搜索与抓取数据与分析

概览

AI 生成的概览

基于 Firecrawl 的网页数据服务器：抓取页面为 Markdown、映射与爬取站点、搜索网页、按结构提取数据并进行深度研究。

可以做什么
  • 抓取单个页面并返回 Markdown 等格式，支持正文提取、等待与页面操作
  • 通过站点地图与链接发现列出站点 URL，可按搜索词过滤
  • 搜索网页，并可选地抓取每条结果页面
  • 启动站点爬取与批量抓取任务，并按任务 id 轮询状态
  • 借助 LLM 按提示词和可选 JSON schema 从页面提取结构化字段
  • 跨多个来源进行深度研究，并为站点生成 llms.txt 文档
适用场景
适合需要可读网页内容或整站数据的场景：把页面转成干净的 Markdown、收集站点 URL、执行搜索、按 schema 抽取结构化字段，或汇总多来源研究结果。
运行要求
通过 npx（npm 包）或 Docker 镜像在本地以 stdio 运行，容器默认进入 MCP 模式。需要 FIRECRAWL_API_KEY 环境变量中的 Firecrawl API 密钥，并能访问 Firecrawl API 的网络。未设置密钥时调用会直接失败，不会发送任何内容。
安装前请注意
需要 Firecrawl API 密钥，该密钥会交给服务器用于上游调用，用量可能由 Firecrawl 计费。抓取的页面内容、URL 和研究查询会发送到 Firecrawl API，提取与深度研究在上游由 LLM 完成。爬取和批量任务异步执行，会消耗 API 配额。

安装

在 SourceWeft 中

  1. 打开 控制台中的 Crawlport，将其添加到工作区。
  2. 为需要使用其工具的对话启用该服务。

Desktop only，通过 STDIO。 STDIO 服务会启动本地进程，因此需要 SourceWeft 桌面宿主。

其他 MCP 客户端

参照 仓库 中的启动说明。

README

crawlport

crawlport is a doorway between an agent and the readable web, backed by Firecrawl. Give it one page and it hands back clean markdown; give it a whole site and it will map the pages, crawl a section, or scrape a list in batch; give it a question and it can search, pull structured fields out of pages to a schema you describe, research across many sources, or draft the llms.txt a site would publish about itself. It runs as a CLI or as an MCP server, and it needs a Firecrawl API key.

Setup

Get a key from Firecrawl and export it:

bash
export FIRECRAWL_API_KEY=<redacted>

Without the variable set, every command and tool call fails with an error that names FIRECRAWL_API_KEY — nothing is sent anywhere.

Install

bash
npx @reneturcios/crawlport scrape https://example.com/some/page

Or run the container image:

bash
docker run -i -e FIRECRAWL_API_KEY ghcr.io/cloudcorprecords/crawlport

The container defaults to the MCP server (see below). Pass CLI arguments to use it as a CLI instead.

CLI

bash
# One page, as markdown (add --formats html,links etc. for more)crawlport scrape https://example.com/article --only-main-content
# Discover a site's URLs, optionally filteredcrawlport map https://example.com --search pricing --limit 100
# Web search, with locale controlscrawlport search "standing desk reviews" --limit 5 --lang en --country US
# Start a crawl, then poll it with the returned job idcrawlport crawl https://example.com/docs --limit 50 --max-depth 2crawlport crawl-status <job-id>
# Structured extraction and batch scrapingcrawlport extract https://example.com/a https://example.com/b --prompt "pull the price"crawlport batch-scrape https://example.com/a https://example.com/bcrawlport batch-status <job-id>
# Research and llms.txtcrawlport deep-research "home battery storage costs in 2026" --max-urls 40crawlport llmstxt https://example.com --max-urls 25
crawlport --version

Successful calls print the API's JSON response. Crawl and batch jobs are asynchronous upstream: the start call returns a job id, and the matching status command reads its progress and results. Exit codes: 0 on success, 1 when the API call fails (including a missing key), 2 for usage errors.

MCP server

One bin serves both faces. The mcp subcommand starts a stdio MCP server:

bash
npx -y @reneturcios/crawlport mcp

MCP client configuration:

json
{  "mcpServers": {    "crawlport": {      "command": "npx",      "args": ["-y", "@reneturcios/crawlport", "mcp"],      "env": { "FIRECRAWL_API_KEY": "your-key-here" }    }  }}

Tools

ToolInputWhat it does
firecrawl_scrapeurl (required); formats, onlyMainContent, waitFor, timeout, actions, includeTags, excludeTags, blockAds, proxy (optional)Scrape one page via POST /v1/scrape.
firecrawl_mapurl (required); search, limit, ignoreSitemapList a site's discovered URLs via POST /v1/map.
firecrawl_searchquery (required); limit, lang, country, scrapeOptionsWeb search via POST /v1/search, optionally scraping each result.
firecrawl_crawlurl (required); limit, maxDepth, allowBackwardLinks, allowExternalLinks, excludePaths, includePaths, scrapeOptionsStart a crawl via POST /v1/crawl; returns a job id.
firecrawl_check_crawl_statusid (required)Poll a crawl job via GET /v1/crawl/{id}.
firecrawl_extracturls (required); prompt, schema, enableWebSearch, allowExternalLinksLLM extraction to a prompt/schema via POST /v1/extract.
firecrawl_batch_scrapeurls (required); formats, onlyMainContentStart a batch scrape via POST /v1/batch/scrape; returns a job id.
firecrawl_check_batch_statusid (required)Poll a batch job via GET /v1/batch/scrape/{id}.
firecrawl_deep_researchquery (required); maxDepth, timeLimit, maxUrlsMulti-source research via POST /v1/deep-research.
firecrawl_generate_llmstxturl (required); maxUrls, showFullTextGenerate llms.txt content via POST /v1/llmstxt.

Tool results are the upstream JSON, pretty-printed as text. Optional arguments are forwarded only when provided, so the API's defaults apply otherwise. Upstream failures come back as tool errors carrying the HTTP status and the API's own message.

Development

bash
npm installnpm run build   # tsc -> dist/npm test        # node test runner over test/*.test.ts, no network (fake fetch fixtures)npm run dev     # run the CLI from source via tsx

License

MIT — see LICENSE.

来源：README.md，提交 8ae8ce6

工具

10

firecrawl_scrape

Scrape a single page and return its content in the requested formats, with options for main-content extraction, waiting, and page actions.

输入参数url必填proxyactionsformatstimeoutwaitForblockAdsexcludeTagsincludeTagsonlyMainContent

firecrawl_map

Discover the URLs reachable from a site, optionally filtered by a search term, using sitemap and link discovery.

输入参数url必填limitsearchignoreSitemap

firecrawl_crawl

Start an asynchronous crawl of a site section and return a job id that can be polled for status.

输入参数url必填limitmaxDepthexcludePathsincludePathsscrapeOptionsallowBackwardLinksallowExternalLinks

firecrawl_check_crawl_status

Check the progress and results of a crawl job started earlier.

输入参数id必填

firecrawl_extract

Extract structured data from a set of pages with an LLM, guided by a prompt and an optional JSON schema.

输入参数urls必填promptschemaenableWebSearchallowExternalLinks

firecrawl_batch_scrape

Scrape many URLs in one batch job and return a job id that can be polled for status.

输入参数urls必填formatsonlyMainContent

firecrawl_check_batch_status

Check the progress and results of a batch scrape job started earlier.

输入参数id必填

firecrawl_deep_research

Research a question across many web sources and return a synthesized analysis with its sources.

输入参数query必填maxUrlsmaxDepthtimeLimit

firecrawl_generate_llmstxt

Generate an llms.txt document for a site, describing its pages in a form LLMs can consume.

输入参数url必填maxUrlsshowFullText

版本历史

1
  1. v0.1.0最新Oct 31, 2026

评分

已满足适用检查项的 90%

该评分衡量此收录条目的完整程度，依据注册中心条目、端点与公开来源计算得出。它不是对服务器本身的评价，也不表示服务器安全或当前可用：可连通性是带日期的检查结果，评分也不包含安装量或用户评价。

  • 近 30 天内可连通25 分
  • 已声明认证方式10 分

    请在注册中心条目中声明凭据请求头，或保持端点能够响应探测，以便判定其认证方式。

  • 已列出工具必需15 分
  • 描述必需15 分
  • 图标10 分
  • 文档15 分
  • 许可证10 分