Crawlport

io.github.CloudCorpRecordsPublished by GitHub account CloudCorpRecordsv0.1.0Updated Nov 3, 2026

Firecrawl web data: scrape, map, search, crawl, extract, research, and llms.txt.

VerifiedSTDIODesktop onlyWeb Search & ScrapingData & Analytics

Overview

AI-generated overview

Firecrawl-backed web data server: scrape pages to markdown, map and crawl sites, search the web, extract structured data, and run deep research.

What you can do
  • Scrape a single page into markdown or other formats, with main-content filtering, waits and page actions
  • Discover a site's URLs via sitemap and link discovery, optionally filtered by a search term
  • Search the web and optionally scrape each result page
  • Start site crawls and batch scrapes, then poll their job status by id
  • Extract structured fields from pages with an LLM using a prompt and optional JSON schema
  • Run multi-source deep research and generate an llms.txt document for a site
When to use it
Useful when an assistant needs readable web content or site-wide data: turning pages into clean markdown, collecting URLs from a site, running searches, pulling structured fields to a schema, or compiling research across many sources.
Requirements
Runs locally over stdio via npx (npm package) or the Docker image; the container defaults to MCP mode. Needs a Firecrawl API key in the FIRECRAWL_API_KEY environment variable, and network access to the Firecrawl API. Without the key, calls fail before anything is sent.
Before you install
Requires a Firecrawl API key, which is passed to the server and used for upstream calls; usage may be billed by Firecrawl. Scraped page content, URLs and research queries are sent to Firecrawl's API, and extraction and deep research are performed upstream with an LLM. Crawls and batch jobs run asynchronously and consume API quota.

Installation

In SourceWeft

  1. Open Crawlport in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Desktop only via STDIO. STDIO servers start a local process, so they need the SourceWeft desktop host.

Other MCP clients

Follow the launch instructions in the repository.

README

crawlport

crawlport is a doorway between an agent and the readable web, backed by Firecrawl. Give it one page and it hands back clean markdown; give it a whole site and it will map the pages, crawl a section, or scrape a list in batch; give it a question and it can search, pull structured fields out of pages to a schema you describe, research across many sources, or draft the llms.txt a site would publish about itself. It runs as a CLI or as an MCP server, and it needs a Firecrawl API key.

Setup

Get a key from Firecrawl and export it:

bash
export FIRECRAWL_API_KEY=<redacted>

Without the variable set, every command and tool call fails with an error that names FIRECRAWL_API_KEY — nothing is sent anywhere.

Install

bash
npx @reneturcios/crawlport scrape https://example.com/some/page

Or run the container image:

bash
docker run -i -e FIRECRAWL_API_KEY ghcr.io/cloudcorprecords/crawlport

The container defaults to the MCP server (see below). Pass CLI arguments to use it as a CLI instead.

CLI

bash
# One page, as markdown (add --formats html,links etc. for more)crawlport scrape https://example.com/article --only-main-content
# Discover a site's URLs, optionally filteredcrawlport map https://example.com --search pricing --limit 100
# Web search, with locale controlscrawlport search "standing desk reviews" --limit 5 --lang en --country US
# Start a crawl, then poll it with the returned job idcrawlport crawl https://example.com/docs --limit 50 --max-depth 2crawlport crawl-status <job-id>
# Structured extraction and batch scrapingcrawlport extract https://example.com/a https://example.com/b --prompt "pull the price"crawlport batch-scrape https://example.com/a https://example.com/bcrawlport batch-status <job-id>
# Research and llms.txtcrawlport deep-research "home battery storage costs in 2026" --max-urls 40crawlport llmstxt https://example.com --max-urls 25
crawlport --version

Successful calls print the API's JSON response. Crawl and batch jobs are asynchronous upstream: the start call returns a job id, and the matching status command reads its progress and results. Exit codes: 0 on success, 1 when the API call fails (including a missing key), 2 for usage errors.

MCP server

One bin serves both faces. The mcp subcommand starts a stdio MCP server:

bash
npx -y @reneturcios/crawlport mcp

MCP client configuration:

json
{  "mcpServers": {    "crawlport": {      "command": "npx",      "args": ["-y", "@reneturcios/crawlport", "mcp"],      "env": { "FIRECRAWL_API_KEY": "your-key-here" }    }  }}

Tools

ToolInputWhat it does
firecrawl_scrapeurl (required); formats, onlyMainContent, waitFor, timeout, actions, includeTags, excludeTags, blockAds, proxy (optional)Scrape one page via POST /v1/scrape.
firecrawl_mapurl (required); search, limit, ignoreSitemapList a site's discovered URLs via POST /v1/map.
firecrawl_searchquery (required); limit, lang, country, scrapeOptionsWeb search via POST /v1/search, optionally scraping each result.
firecrawl_crawlurl (required); limit, maxDepth, allowBackwardLinks, allowExternalLinks, excludePaths, includePaths, scrapeOptionsStart a crawl via POST /v1/crawl; returns a job id.
firecrawl_check_crawl_statusid (required)Poll a crawl job via GET /v1/crawl/{id}.
firecrawl_extracturls (required); prompt, schema, enableWebSearch, allowExternalLinksLLM extraction to a prompt/schema via POST /v1/extract.
firecrawl_batch_scrapeurls (required); formats, onlyMainContentStart a batch scrape via POST /v1/batch/scrape; returns a job id.
firecrawl_check_batch_statusid (required)Poll a batch job via GET /v1/batch/scrape/{id}.
firecrawl_deep_researchquery (required); maxDepth, timeLimit, maxUrlsMulti-source research via POST /v1/deep-research.
firecrawl_generate_llmstxturl (required); maxUrls, showFullTextGenerate llms.txt content via POST /v1/llmstxt.

Tool results are the upstream JSON, pretty-printed as text. Optional arguments are forwarded only when provided, so the API's defaults apply otherwise. Upstream failures come back as tool errors carrying the HTTP status and the API's own message.

Development

bash
npm installnpm run build   # tsc -> dist/npm test        # node test runner over test/*.test.ts, no network (fake fetch fixtures)npm run dev     # run the CLI from source via tsx

License

MIT — see LICENSE.

Source: README.md at commit 8ae8ce6

Tools

10

firecrawl_scrape

Scrape a single page and return its content in the requested formats, with options for main-content extraction, waiting, and page actions.

ParametersurlrequiredproxyactionsformatstimeoutwaitForblockAdsexcludeTagsincludeTagsonlyMainContent

firecrawl_map

Discover the URLs reachable from a site, optionally filtered by a search term, using sitemap and link discovery.

ParametersurlrequiredlimitsearchignoreSitemap

firecrawl_crawl

Start an asynchronous crawl of a site section and return a job id that can be polled for status.

ParametersurlrequiredlimitmaxDepthexcludePathsincludePathsscrapeOptionsallowBackwardLinksallowExternalLinks

firecrawl_check_crawl_status

Check the progress and results of a crawl job started earlier.

Parametersidrequired

firecrawl_extract

Extract structured data from a set of pages with an LLM, guided by a prompt and an optional JSON schema.

ParametersurlsrequiredpromptschemaenableWebSearchallowExternalLinks

firecrawl_batch_scrape

Scrape many URLs in one batch job and return a job id that can be polled for status.

ParametersurlsrequiredformatsonlyMainContent

firecrawl_check_batch_status

Check the progress and results of a batch scrape job started earlier.

Parametersidrequired

firecrawl_deep_research

Research a question across many web sources and return a synthesized analysis with its sources.

ParametersqueryrequiredmaxUrlsmaxDepthtimeLimit

firecrawl_generate_llmstxt

Generate an llms.txt document for a site, describing its pages in a form LLMs can consume.

ParametersurlrequiredmaxUrlsshowFullText

Version history

1
  1. v0.1.0LatestOct 31, 2026

Score

90% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 days25 pts
  • Auth declared10 pts

    Declare the credential headers in the registry entry, or keep the endpoint answering the probe so its auth can be classified.

  • Tools listedRequired15 pts
  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts