Website to Markdown Crawler

io.github.prometheus-maker由 GitHub 账号 prometheus-maker 发布v1.0.0更新于 Nov 7, 2026

Crawl a website into clean Markdown for LLMs/RAG: main content only, linked docs converted.

已验证Streamable HTTP可网页运行网页搜索与抓取知识与记忆

概览

AI 生成的概览

由 Apify 托管的 MCP 服务器，可爬取网站并输出仅含正文的干净 Markdown，同时转换链接的文档，供 LLM 与 RAG 使用。

可以做什么
  • 从给定网址开始爬取站点，返回仅含正文的 Markdown 页面
  • 在爬取过程中把链接的 PDF、Word、Excel 文档转换为 Markdown
  • 可控制爬取范围、最大页数、是否保留链接以及可选的文本分块
  • 通过 Apify 托管的 MCP 端点调用，同一工具也可作为 Apify API actor 使用
适用场景
适合需要把文档站点或网站内容转成 Markdown 以构建 RAG 索引、知识库或 LLM 上下文，并且希望使用托管爬虫而不自行搭建抓取流程的场景。
运行要求
远程 streamable HTTP 端点 mcp.apify.com；需要 Apify 账号与 API 令牌，通过 Authorization 的 Bearer 头传递；需要网络访问；按页计费。
安装前请注意
需要 Apify API 令牌，这是应妥善保密的付费凭证。按使用量计费，每爬取一页都会产生费用，价格随 Apify 套餐而异。爬取会把目标网址发送到 Apify 服务器；README 还列出 OCR、SEO 审计、截图、转写等其他付费工具，可能在同一端点上启用。

安装

在 SourceWeft 中

  1. 打开 控制台中的 Website to Markdown Crawler，将其添加到工作区。
  2. 连接 所需的 凭据。它们会被私密存储，绝不会在公开页面上展示。
  3. 为需要使用其工具的对话启用该服务。

Web executable，通过 Streamable HTTP。 远程服务在工作区中配置后即可从网页运行时运行。

交给智能体安装

把这段提示词粘贴到 Claude Code、Codex、Cursor 等智能体，或 SourceWeft 对话中。智能体会读取该服务器的安装指南，先向你展示发布者、端点和登录方式，经你同意后再把它添加到你的客户端。密钥不会写进任何命令。

阅读 https://sourceweft.com/mcp/io.github.prometheus-maker%2Fwebsite-to-markdown/install.md，并按其中说明把这个 MCP 服务器接入我的客户端。修改任何配置之前，先告诉我它的发布者、端点和登录方式，等我确认。不要把 API 密钥写进命令或我会提交的文件；问我密钥应从哪里获取。

查看智能体所依据的安装指南

其他 MCP 客户端

把它添加到你客户端的 mcpServers 配置中。 客户端会在首次连接时完成 OAuth 登录。

{
  "mcpServers": {
    "website-to-markdown-crawler": {
      "type": "http",
      "url": "https://mcp.apify.com/?tools=prometheus-maker/website-to-markdown-crawler"
    }
  }
}

README

Apify tools: PDF to Markdown, OCR, Website to Markdown crawler, SEO audit, Tech Stack

Working examples for my pay-per-use tools on the Apify Store. Each one is an API: send URLs, get JSON back. You only pay for what you process, and there's no subscription.

ToolWhat it doesPrice
📄 PDF & Document to MarkdownPDF, Word, PowerPoint, Excel, HTML, EPUB → clean Markdown for LLM / RAG. Tables, two-column papers, OCR for scans, chunking$0.002 per document + $0.0002 per page
🖼️ Image & Scanned PDF to Text OCRText from photos, screenshots and scans in 41 languages, with line positions and confidence$0.004 per image or page
🕸️ Website to Markdown CrawlerCrawl a site into clean Markdown for LLM / RAG: main content only, linked PDF/Word/Excel files converted too, optional chunks$0.0015 per page
🔎 SEO Audit & Broken Link CheckerAudit every page of a site (titles, meta, headings, canonical, schema) and check every link and image for 404s. 0-100 score per page$0.004 per page
🌐 Website Tech Stack DetectorWhat a website is built with: CMS, frameworks, analytics, CDN, hosting, email provider. 7,000+ technologies (BuiltWith / Wappalyzer alternative)$0.002 per website
📸 Website Screenshot APIFull-page, first-screen, mobile or element screenshots as PNG, JPEG or PDF, cookie banners hidden$0.005 per full page, $0.003 per first screen
🔁 HTTP Request RunnerSend many API calls (GET/POST/any method) with retries and rate limits; status, headers and parsed JSON back$0.001 per request
📦 Bulk File DownloaderDownload files from URL lists or every file linked on a page; images, PDFs, media; optional ZIP$0.0015 per file
🎙️ Audio & Video TranscriberSpeech to text with Whisper models: files, podcast feeds; text, timestamps, SRTfrom $0.003 per audio minute

Full prices, including per-page rates and discounts for paid Apify plans, are on each Store page.

Quick start (curl)

Get an API token from Apify Console → Integrations, then:

bash
curl -X POST "https://api.apify.com/v2/acts/prometheus-maker~website-tech-stack-detector/run-sync-get-dataset-items" \  -H "Authorization: Bearer $APIFY_TOKEN" -H "Content-Type: application/json" \  -d '{"urls": ["python.org"]}'

The same endpoint works for the other tools: replace the tool name and the input.

Use them from Claude, Cursor or any MCP client

Every tool is also an MCP server, hosted by Apify (streamable HTTP). Add one to your client with your Apify token:

json
{  "mcpServers": {    "pdf-to-markdown": {      "url": "https://mcp.apify.com/?tools=prometheus-maker/pdf-document-to-markdown",      "headers": { "Authorization": "Bearer YOUR_APIFY_TOKEN" }    }  }}

Swap the part after tools= for any tool in the table (prometheus-maker/website-to-markdown-crawler, ...), or list several separated by commas. The same entries are in the official MCP Registry under io.github.prometheus-maker/* (files in mcp/). You pay per use, as with the API.

Python

bash
pip install apify-clientexport APIFY_TOKEN=...python python/pdf_to_markdown.py      # PDF → Markdown + RAG chunkspython python/image_to_text_ocr.py    # image → textpython python/tech_stack.py           # domains → technologiespython python/website_to_markdown.py  # website → Markdown pages + chunkspython python/seo_audit.py            # website → SEO scores, issues, broken links

The scripts are in python/. They use apify-client 3.x.

Inputs at a glance

ToolMain inputUseful options
PDF to MarkdownstartUrls: [{"url": ...}]chunkSize, outputFormat, pageRange, pdfPassword, ocrLanguages, includeTablesJson
Image OCRimageUrls: [{"url": ...}]languages (e.g. ["eng","deu"]), engine, includeLines (line positions)
Tech stackurls: ["example.com", ...]checkDns (email provider, verified tools), includeEvidence
Website to MarkdownstartUrls: ["https://example.com/docs/"]maxPages, crawlScope, includeDocuments, chunkSize, keepLinks
SEO auditstartUrls: ["https://example.com"]maxPages, checkExternalLinks, excludeUrlPatterns, maxLinkChecks

See each Store page for every field and a full output example.

About

These tools are built and maintained by Prometheus (notes). Issues and requests are welcome in this repo.

来源：README.md，提交 c6aad2a

工具

0
该服务器需要登录，因此 SourceWeft 无法在没有账号的情况下读取其工具，也未收录。已连接账号所能看到的内容可能因账号及其权限而异。

版本历史

1
  1. v1.0.0最新Nov 7, 2026

评分

已满足适用检查项的 85%

该评分衡量此收录条目的完整程度，依据注册中心条目、端点与公开来源计算得出。它不是对服务器本身的评价，也不表示服务器安全或当前可用：可连通性是带日期的检查结果，评分也不包含安装量或用户评价。

  • 近 30 天内可连通必需25 分
  • 已声明认证方式必需10 分
  • 已列出工具15 分

    请让服务器在无凭据时响应 tools/list，或发布可在沙箱中无凭据启动的软件包。

  • 描述必需15 分
  • 图标10 分
  • 文档15 分
  • 许可证10 分