agent-web — URL to LLM-ready markdown: a polite, robots-respecting web page reader (free)

io.github.foomworks由 GitHub 帳號 foomworks 發布v0.2.1更新於 Oct 24, 2026

URL to clean markdown for LLMs: a polite, robots.txt-respecting web reader. Free, no API key

已驗證Streamable HTTP可網頁執行網頁搜尋與抓取知識與記憶

概覽

AI 產生的概覽

遠端 MCP 伺服器，抓取公開網址並回傳乾淨、適合大型語言模型使用的 Markdown，遵守 robots.txt 且僅供讀取。

可以做什麼
  • 抓取單一公開網址，回傳標題、描述、字數與完整 Markdown
  • 以低成本預覽頁面，只回傳描述與前約 600 個字元
  • 要求網頁的 PNG 截圖或 PDF 渲染（付費，目前無法使用）
  • 用免費探測工具檢查渲染通道狀態
適用情境
當助理需要把文章、文件頁等靜態 HTML 頁面讀成乾淨文字而非原始 HTML，且希望讀取端遵守 robots.txt、不繞過付費牆或反機器人機制時適用。
執行需求
遠端 streamable HTTP 端點；未宣告需要帳號、API 金鑰、標頭或環境變數。僅支援公開可存取的網址，私有、回環與內網主機會被封鎖。僅支援靜態 HTML，內文上限約 2 MB，抓取逾時約 12 秒。
安裝前請注意
截圖與 PDF 工具標示為透過 x402 付費，且在瀏覽器渲染開通前會回傳 503；服務方表示本身不持有金鑰、不會發起付款。抓取的頁面內容會傳送給第三方營運方，且你需對所提交網址擁有抓取權利。

安裝

在 SourceWeft 中

  1. 開啟 儀表板中的 agent-web — URL to LLM-ready markdown: a polite, robots-respecting web page reader (free)，將其新增到工作區。
  2. 為需要使用其工具的對話啟用該服務。

Web executable，透過 Streamable HTTP。 遠端服務在工作區中設定後即可從網頁執行環境執行。

交給代理程式安裝

把這段提示詞貼上到 Claude Code、Codex、Cursor 等代理程式，或 SourceWeft 對話中。代理程式會讀取該伺服器的安裝指南，先向你展示發布者、端點和登入方式，經你同意後再把它新增到你的客戶端。金鑰不會寫進任何命令。

閱讀 https://sourceweft.com/mcp/io.github.foomworks%2Fagent-web/install.md，並按其中說明把這個 MCP 伺服器接入我的客戶端。修改任何設定之前，先告訴我它的發布者、端點和登入方式，等我確認。不要把 API 金鑰寫進命令或我會提交的檔案；問我金鑰應從哪裡取得。

檢視代理程式所依據的安裝指南

其他 MCP 客戶端

把它新增到你客戶端的 mcpServers 設定中。

{
  "mcpServers": {
    "agent-web-url-to-llm-ready-markdown-a-po": {
      "type": "http",
      "url": "https://agent-web.foomworks.workers.dev/mcp"
    }
  }
}

README

agent-web

URL → LLM-ready markdown. A charter-clean, polite, robots-respecting web reader for AI agents — honors robots.txt, identifies honestly, and never bypasses anti-bot/CAPTCHA or paywalls. No account, no API key — free per-call access.

🌐 Live: https://agent-web.foomworks.workers.dev · 🔌 MCP: https://agent-web.foomworks.workers.dev/mcp

What it gives an agent

Give it one publicly reachable URL; get back the page as clean, LLM-ready markdown — title, word count, and the text — without parsing raw HTML yourself. Server-rendered (static) HTML pages. Ideal for summarizing an article, extracting docs, or feeding a page into an LLM.

Connect as an MCP server (recommended)

Streamable HTTP, JSON-RPC 2.0:

https://agent-web.foomworks.workers.dev/mcp
ToolCostReturns
read_urlfreefetch a URL → full LLM-ready markdown (title, word count, markdown)
read_url_previewfreethe first ~600 characters of the markdown — a cheap look before the full pull

Or call it over HTTP

bash
BASE=https://agent-web.foomworks.workers.devcurl -s "$BASE/read?url=https://example.com/&ref=readme"          # full markdowncurl -s "$BASE/read/preview?url=https://example.com/&ref=readme"  # first ~600 chars

Discovery: /.well-known/mcp.json (MCP) · /.well-known/agent-card.json (A2A) · /openapi.json · /policy (Acceptable-Use Policy)

Charter-clean by design (the bright line)

This is a polite reader, not a scraper:

  • Honors the origin's robots.txt for our user-agent — a disallowed path is refused without fetching; an unconfirmable robots.txt (5xx/error) is treated conservatively as disallowed.
  • Identifies honestly with a descriptive user-agent — never spoofs a browser.
  • Read-only GET of the single URL you supply — no crawl, no proxy rotation.
  • Never bypasses anti-bot/CAPTCHA, paywalls, or login walls.
  • SSRF-guarded: private/loopback/link-local/internal hosts are blocked and every redirect hop is re-validated.

You are responsible for your right to fetch the URL you submit.

Limits

  • Static HTML only for now — JavaScript-rendered pages, screenshots, and PDF are a later increment.
  • Body cap ~2 MB, fetch timeout ~12 s.
  • The service holds no keys and never initiates payments.

Install the AgentSkill

SKILL.md is a portable AgentSkill (ClawHub / skills.sh / Claude Skills).

About

agent-web is operated by FOOM — an AI-operated, human-supervised autonomous service. License: MIT.

來源：README.md，提交 24b8d20

工具

5

read_url

Fetch one publicly reachable URL and return clean, LLM-ready markdown (title + page description + word count + markdown). HTML pages are extracted to markdown — including HTML tables, which become GitHub-flavored Markdown tables; URLs pointing straight at a Markdown or plain-text document (raw READMEs, llms.txt, docs) are passed through verbatim. Polite by design: honors the origin's robots.txt for our user-agent, identifies honestly, read-only GET, never bypasses anti-bot/CAPTCHA/paywalls. Free. JavaScript-rendered pages are not supported yet.

輸入參數url必填

read_url_preview

Like read_url but returns only the page description + the first ~600 characters of the markdown (title + word count + a truncation flag) — a cheap way to check a page's relevance before pulling the full text. Free.

輸入參數url必填

render_preview

Free discovery stub for the screenshot/PDF render lane. Same robots/SSRF guards as read_url. Currently returns a 503 (no charge) until Cloudflare Browser Rendering is provisioned on this account.

輸入參數url必填

render_screenshot

PAID (x402): returns x402 payment instructions for a PNG screenshot of a publicly reachable URL, rendered via a real, robots-respecting headless browser. Use render_preview (free) first. Currently soft-skips 503 until Cloudflare Browser Rendering is provisioned.

輸入參數url必填widthfullPage

render_pdf

PAID (x402): returns x402 payment instructions for a PDF render of a publicly reachable URL, via a real, robots-respecting headless browser. Currently soft-skips 503 until Cloudflare Browser Rendering is provisioned.

輸入參數url必填

版本歷史

1
  1. v0.2.1最新Sep 16, 2026

評分

已滿足適用檢查項的 100%

該評分衡量此收錄條目的完整程度，依據註冊中心條目、端點與公開來源計算得出。它不是對伺服器本身的評價，也不表示伺服器安全或目前可用：可連通性是帶日期的檢查結果，評分也不包含安裝量或使用者評價。

  • 近 30 天內可連通必需25 分
  • 已宣告認證方式必需10 分
  • 已列出工具必需15 分
  • 描述必需15 分
  • 圖示10 分
  • 文件15 分
  • 授權條款10 分