Crawlport

io.github.CloudCorpRecords由 GitHub 帳號 CloudCorpRecords 發布v0.1.0更新於 Nov 3, 2026

Firecrawl web data: scrape, map, search, crawl, extract, research, and llms.txt.

已驗證STDIO僅桌面網頁搜尋與抓取資料與分析

概覽

AI 產生的概覽

以 Firecrawl 為後端的網頁資料伺服器：擷取頁面為 Markdown、對網站做對應與爬取、搜尋網頁、依結構擷取資料並進行深度研究。

可以做什麼
  • 擷取單一頁面並以 Markdown 等格式回傳，支援主內容擷取、等待與頁面操作
  • 透過網站地圖與連結探索列出網站 URL，可依搜尋詞篩選
  • 搜尋網頁，並可選擇擷取每一筆結果頁面
  • 啟動網站爬取與批次擷取工作，並依工作 id 輪詢狀態
  • 透過 LLM 依提示詞與選用的 JSON schema 從頁面擷取結構化欄位
  • 跨多個來源進行深度研究，並為網站產生 llms.txt 文件
適用情境
適合需要可讀網頁內容或整站資料的情境：把頁面轉成乾淨的 Markdown、收集網站 URL、執行搜尋、依 schema 抽取結構化欄位，或彙整多來源的研究結果。
執行需求
透過 npx（npm 套件）或 Docker 映像檔在本機以 stdio 執行，容器預設進入 MCP 模式。需要 FIRECRAWL_API_KEY 環境變數中的 Firecrawl API 金鑰，以及可連線至 Firecrawl API 的網路。未設定金鑰時呼叫會直接失敗，不會送出任何內容。
安裝前請注意
需要 Firecrawl API 金鑰，該金鑰會交給伺服器用於上游呼叫，用量可能由 Firecrawl 計費。擷取的頁面內容、URL 與研究查詢會傳送到 Firecrawl API，擷取與深度研究在上游由 LLM 完成。爬取與批次工作以非同步方式執行，會消耗 API 配額。

安裝

在 SourceWeft 中

  1. 開啟 儀表板中的 Crawlport，將其新增到工作區。
  2. 為需要使用其工具的對話啟用該服務。

Desktop only，透過 STDIO。 STDIO 服務會啟動本機處理程序，因此需要 SourceWeft 桌面主機。

其他 MCP 客戶端

參照 儲存庫 中的啟動說明。

README

crawlport

crawlport is a doorway between an agent and the readable web, backed by Firecrawl. Give it one page and it hands back clean markdown; give it a whole site and it will map the pages, crawl a section, or scrape a list in batch; give it a question and it can search, pull structured fields out of pages to a schema you describe, research across many sources, or draft the llms.txt a site would publish about itself. It runs as a CLI or as an MCP server, and it needs a Firecrawl API key.

Setup

Get a key from Firecrawl and export it:

bash
export FIRECRAWL_API_KEY=<redacted>

Without the variable set, every command and tool call fails with an error that names FIRECRAWL_API_KEY — nothing is sent anywhere.

Install

bash
npx @reneturcios/crawlport scrape https://example.com/some/page

Or run the container image:

bash
docker run -i -e FIRECRAWL_API_KEY ghcr.io/cloudcorprecords/crawlport

The container defaults to the MCP server (see below). Pass CLI arguments to use it as a CLI instead.

CLI

bash
# One page, as markdown (add --formats html,links etc. for more)crawlport scrape https://example.com/article --only-main-content
# Discover a site's URLs, optionally filteredcrawlport map https://example.com --search pricing --limit 100
# Web search, with locale controlscrawlport search "standing desk reviews" --limit 5 --lang en --country US
# Start a crawl, then poll it with the returned job idcrawlport crawl https://example.com/docs --limit 50 --max-depth 2crawlport crawl-status <job-id>
# Structured extraction and batch scrapingcrawlport extract https://example.com/a https://example.com/b --prompt "pull the price"crawlport batch-scrape https://example.com/a https://example.com/bcrawlport batch-status <job-id>
# Research and llms.txtcrawlport deep-research "home battery storage costs in 2026" --max-urls 40crawlport llmstxt https://example.com --max-urls 25
crawlport --version

Successful calls print the API's JSON response. Crawl and batch jobs are asynchronous upstream: the start call returns a job id, and the matching status command reads its progress and results. Exit codes: 0 on success, 1 when the API call fails (including a missing key), 2 for usage errors.

MCP server

One bin serves both faces. The mcp subcommand starts a stdio MCP server:

bash
npx -y @reneturcios/crawlport mcp

MCP client configuration:

json
{  "mcpServers": {    "crawlport": {      "command": "npx",      "args": ["-y", "@reneturcios/crawlport", "mcp"],      "env": { "FIRECRAWL_API_KEY": "your-key-here" }    }  }}

Tools

ToolInputWhat it does
firecrawl_scrapeurl (required); formats, onlyMainContent, waitFor, timeout, actions, includeTags, excludeTags, blockAds, proxy (optional)Scrape one page via POST /v1/scrape.
firecrawl_mapurl (required); search, limit, ignoreSitemapList a site's discovered URLs via POST /v1/map.
firecrawl_searchquery (required); limit, lang, country, scrapeOptionsWeb search via POST /v1/search, optionally scraping each result.
firecrawl_crawlurl (required); limit, maxDepth, allowBackwardLinks, allowExternalLinks, excludePaths, includePaths, scrapeOptionsStart a crawl via POST /v1/crawl; returns a job id.
firecrawl_check_crawl_statusid (required)Poll a crawl job via GET /v1/crawl/{id}.
firecrawl_extracturls (required); prompt, schema, enableWebSearch, allowExternalLinksLLM extraction to a prompt/schema via POST /v1/extract.
firecrawl_batch_scrapeurls (required); formats, onlyMainContentStart a batch scrape via POST /v1/batch/scrape; returns a job id.
firecrawl_check_batch_statusid (required)Poll a batch job via GET /v1/batch/scrape/{id}.
firecrawl_deep_researchquery (required); maxDepth, timeLimit, maxUrlsMulti-source research via POST /v1/deep-research.
firecrawl_generate_llmstxturl (required); maxUrls, showFullTextGenerate llms.txt content via POST /v1/llmstxt.

Tool results are the upstream JSON, pretty-printed as text. Optional arguments are forwarded only when provided, so the API's defaults apply otherwise. Upstream failures come back as tool errors carrying the HTTP status and the API's own message.

Development

bash
npm installnpm run build   # tsc -> dist/npm test        # node test runner over test/*.test.ts, no network (fake fetch fixtures)npm run dev     # run the CLI from source via tsx

License

MIT — see LICENSE.

來源：README.md，提交 8ae8ce6

工具

10

firecrawl_scrape

Scrape a single page and return its content in the requested formats, with options for main-content extraction, waiting, and page actions.

輸入參數url必填proxyactionsformatstimeoutwaitForblockAdsexcludeTagsincludeTagsonlyMainContent

firecrawl_map

Discover the URLs reachable from a site, optionally filtered by a search term, using sitemap and link discovery.

輸入參數url必填limitsearchignoreSitemap

firecrawl_crawl

Start an asynchronous crawl of a site section and return a job id that can be polled for status.

輸入參數url必填limitmaxDepthexcludePathsincludePathsscrapeOptionsallowBackwardLinksallowExternalLinks

firecrawl_check_crawl_status

Check the progress and results of a crawl job started earlier.

輸入參數id必填

firecrawl_extract

Extract structured data from a set of pages with an LLM, guided by a prompt and an optional JSON schema.

輸入參數urls必填promptschemaenableWebSearchallowExternalLinks

firecrawl_batch_scrape

Scrape many URLs in one batch job and return a job id that can be polled for status.

輸入參數urls必填formatsonlyMainContent

firecrawl_check_batch_status

Check the progress and results of a batch scrape job started earlier.

輸入參數id必填

firecrawl_deep_research

Research a question across many web sources and return a synthesized analysis with its sources.

輸入參數query必填maxUrlsmaxDepthtimeLimit

firecrawl_generate_llmstxt

Generate an llms.txt document for a site, describing its pages in a form LLMs can consume.

輸入參數url必填maxUrlsshowFullText

版本歷史

1
  1. v0.1.0最新Oct 31, 2026

評分

已滿足適用檢查項的 90%

該評分衡量此收錄條目的完整程度，依據註冊中心條目、端點與公開來源計算得出。它不是對伺服器本身的評價，也不表示伺服器安全或目前可用：可連通性是帶日期的檢查結果，評分也不包含安裝量或使用者評價。

  • 近 30 天內可連通25 分
  • 已宣告認證方式10 分

    請在註冊中心條目中宣告憑證請求標頭，或保持端點能夠回應探測，以便判定其認證方式。

  • 已列出工具必需15 分
  • 描述必需15 分
  • 圖示10 分
  • 文件15 分
  • 授權條款10 分