Website to Markdown Crawler

io.github.prometheus-makerPublished by GitHub account prometheus-makerv1.0.0Updated Nov 7, 2026

Crawl a website into clean Markdown for LLMs/RAG: main content only, linked docs converted.

VerifiedStreamable HTTPWeb executableWeb Search & ScrapingKnowledge & Memory

Overview

AI-generated overview

Apify-hosted MCP server that crawls a website and returns clean Markdown of its main content, plus converted linked documents, for LLM and RAG use.

What you can do
  • Crawl a site starting from given URLs and return main-content-only Markdown pages
  • Convert linked PDF, Word and Excel documents into Markdown as part of the crawl
  • Control crawl scope, maximum page count, link retention and optional text chunking
  • Run through Apify's hosted MCP endpoint, with the same tool also callable as an Apify API actor
When to use it
Useful when you want to turn documentation or a website into Markdown for a RAG index, knowledge base or LLM context, and prefer a hosted crawler over running your own scraping stack.
Requirements
Remote streamable HTTP endpoint at mcp.apify.com; an Apify account and API token sent as an Authorization Bearer header; network access; usage is billed per page.
Before you install
Requires an Apify API token, a paid credential that should be kept secret. Usage is pay-per-use and costs money per crawled page, with prices varying by Apify plan. Crawling sends target URLs to Apify's servers, and the README also lists other paid tools (OCR, SEO audit, screenshots, transcription) that may be enabled on the same endpoint.

Installation

In SourceWeft

  1. Open Website to Markdown Crawler in the dashboard and add it to a workspace.
  2. Connect the required credentials. They are stored privately and never shown on public pages.
  3. Enable the server for the chats that should use its tools.

Web executable via Streamable HTTP. Remote servers run from the web runtime once configured in a workspace.

Use with an agent

Paste this prompt into Claude Code, Codex, Cursor or another agent — or into SourceWeft chat. The agent reads this server's install guide, shows you its publisher, endpoint and sign-in, and adds it to your client once you agree. Keys never go into a command.

Read https://sourceweft.com/mcp/io.github.prometheus-maker%2Fwebsite-to-markdown/install.md and connect the MCP server it describes to my client. Before changing any config, show me its publisher, endpoint and sign-in method and wait for my OK. Never put an API key in a command or in a file I would commit; ask me where it should come from.

Read the install guide the agent follows

Other MCP clients

Add this to your client's mcpServers config. The client completes OAuth sign-in on first connect.

{
  "mcpServers": {
    "website-to-markdown-crawler": {
      "type": "http",
      "url": "https://mcp.apify.com/?tools=prometheus-maker/website-to-markdown-crawler"
    }
  }
}

README

Apify tools: PDF to Markdown, OCR, Website to Markdown crawler, SEO audit, Tech Stack

Working examples for my pay-per-use tools on the Apify Store. Each one is an API: send URLs, get JSON back. You only pay for what you process, and there's no subscription.

ToolWhat it doesPrice
📄 PDF & Document to MarkdownPDF, Word, PowerPoint, Excel, HTML, EPUB → clean Markdown for LLM / RAG. Tables, two-column papers, OCR for scans, chunking$0.002 per document + $0.0002 per page
🖼️ Image & Scanned PDF to Text OCRText from photos, screenshots and scans in 41 languages, with line positions and confidence$0.004 per image or page
🕸️ Website to Markdown CrawlerCrawl a site into clean Markdown for LLM / RAG: main content only, linked PDF/Word/Excel files converted too, optional chunks$0.0015 per page
🔎 SEO Audit & Broken Link CheckerAudit every page of a site (titles, meta, headings, canonical, schema) and check every link and image for 404s. 0-100 score per page$0.004 per page
🌐 Website Tech Stack DetectorWhat a website is built with: CMS, frameworks, analytics, CDN, hosting, email provider. 7,000+ technologies (BuiltWith / Wappalyzer alternative)$0.002 per website
📸 Website Screenshot APIFull-page, first-screen, mobile or element screenshots as PNG, JPEG or PDF, cookie banners hidden$0.005 per full page, $0.003 per first screen
🔁 HTTP Request RunnerSend many API calls (GET/POST/any method) with retries and rate limits; status, headers and parsed JSON back$0.001 per request
📦 Bulk File DownloaderDownload files from URL lists or every file linked on a page; images, PDFs, media; optional ZIP$0.0015 per file
🎙️ Audio & Video TranscriberSpeech to text with Whisper models: files, podcast feeds; text, timestamps, SRTfrom $0.003 per audio minute

Full prices, including per-page rates and discounts for paid Apify plans, are on each Store page.

Quick start (curl)

Get an API token from Apify Console → Integrations, then:

bash
curl -X POST "https://api.apify.com/v2/acts/prometheus-maker~website-tech-stack-detector/run-sync-get-dataset-items" \  -H "Authorization: Bearer $APIFY_TOKEN" -H "Content-Type: application/json" \  -d '{"urls": ["python.org"]}'

The same endpoint works for the other tools: replace the tool name and the input.

Use them from Claude, Cursor or any MCP client

Every tool is also an MCP server, hosted by Apify (streamable HTTP). Add one to your client with your Apify token:

json
{  "mcpServers": {    "pdf-to-markdown": {      "url": "https://mcp.apify.com/?tools=prometheus-maker/pdf-document-to-markdown",      "headers": { "Authorization": "Bearer YOUR_APIFY_TOKEN" }    }  }}

Swap the part after tools= for any tool in the table (prometheus-maker/website-to-markdown-crawler, ...), or list several separated by commas. The same entries are in the official MCP Registry under io.github.prometheus-maker/* (files in mcp/). You pay per use, as with the API.

Python

bash
pip install apify-clientexport APIFY_TOKEN=...python python/pdf_to_markdown.py      # PDF → Markdown + RAG chunkspython python/image_to_text_ocr.py    # image → textpython python/tech_stack.py           # domains → technologiespython python/website_to_markdown.py  # website → Markdown pages + chunkspython python/seo_audit.py            # website → SEO scores, issues, broken links

The scripts are in python/. They use apify-client 3.x.

Inputs at a glance

ToolMain inputUseful options
PDF to MarkdownstartUrls: [{"url": ...}]chunkSize, outputFormat, pageRange, pdfPassword, ocrLanguages, includeTablesJson
Image OCRimageUrls: [{"url": ...}]languages (e.g. ["eng","deu"]), engine, includeLines (line positions)
Tech stackurls: ["example.com", ...]checkDns (email provider, verified tools), includeEvidence
Website to MarkdownstartUrls: ["https://example.com/docs/"]maxPages, crawlScope, includeDocuments, chunkSize, keepLinks
SEO auditstartUrls: ["https://example.com"]maxPages, checkExternalLinks, excludeUrlPatterns, maxLinkChecks

See each Store page for every field and a full output example.

About

These tools are built and maintained by Prometheus (notes). Issues and requests are welcome in this repo.

Source: README.md at commit c6aad2a

Tools

0
This server asks you to sign in, so SourceWeft could not read its tools without an account and has not indexed them. What a connected account sees can depend on that account and its permissions.

Version history

1
  1. v1.0.0LatestNov 7, 2026

Score

85% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 daysRequired25 pts
  • Auth declaredRequired10 pts
  • Tools listed15 pts

    Answer tools/list without a credential, or ship a package that starts in a sandbox without one.

  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts