Sitemap URL Extractor for AI Agents

io.github.industrial-platform-aiv1.0.2更新於 Sep 30, 2026

x402 sitemap discovery and URL extraction for agents. $0.0001 per emitted URL.

已驗證Streamable HTTP可網頁執行Web Search & ScrapingData & Analytics

概覽

AI 產生的概覽

讓助理探索並展開公開的網站地圖，輸出乾淨的 URL 資料列，並依輸出的 URL 計費。

功能
這個託管 MCP 端點提供 Sitemap Intelligence 服務，可探索網站地圖並將其展開為乾淨的 URL 資料列。它是透過同一個 Apify 託管端點存取的數個付費 Industrial Platform 服務之一，工具探索免費，只有在呼叫付費 Actor 時才收費。結果以結構化的機器可讀格式回傳。
適用情境
當助理需要從網站的網站地圖取得 URL 清單，用於爬取、索引或研究流程，且委外這項擷取比自行建置更簡單時使用。它適合已在呼叫其他 Industrial Platform 服務的代理工作流程。
執行需求
需要一個指向託管 Apify 端點的遠端 MCP 用戶端，不需要本機套件。付費執行需要受支援的身分驗證或代理付款方式，例如 Apify OAuth、Apify API 權杖，或使用已儲值錢包的 x402 付款。需要能連線至該端點的網路。
安裝前請注意
呼叫依輸出的 URL 計費，成本會隨回傳的 URL 數量增加；執行前請確認目前價格與資格。付費執行需要憑證或錢包資金，README 提醒不要提交 API 權杖、錢包憑證、機密或私鑰。此服務會抓取任意公開 URL，請了解它對私有或本機網路目標的處理方式。

安裝

在 SourceWeft 中

  1. 開啟 儀表板中的 Sitemap URL Extractor for AI Agents，將其新增到工作區。
  2. 為需要使用其工具的對話啟用該服務。

Web executable，透過 Streamable HTTP。 遠端服務在工作區中設定後即可從網頁執行環境執行。

其他 MCP 客戶端

把它新增到你客戶端的 mcpServers 設定中。

{
  "mcpServers": {
    "sitemap": {
      "type": "http",
      "url": "https://mcp.apify.com?tools=industrial_platform/sitemap-intelligence&payment=x402"
    }
  }
}

README

Industrial Platform Agent Tools

Open-source integration toolkit for connecting AI agents, MCP clients, autonomous workflows, agent orchestrators, and developer applications to Industrial Platform machine-callable services.

Industrial Platform provides specialized paid utilities that other agents can discover and invoke instead of rebuilding research, monitoring, extraction, and document-processing capabilities inside every workflow.

One MCP endpoint for the full portfolio

text
https://mcp.apify.com?tools=industrial_platform/research-brief-agent,industrial_platform/web-change-intelligence,industrial_platform/web-metadata-intelligence,industrial_platform/article-content-intelligence,industrial_platform/pdf-text-intelligence,industrial_platform/sitemap-intelligence&payment=x402

This hosted MCP endpoint exposes the six currently runnable Industrial Platform services through one direct x402 integration. Tool discovery is free; autonomous callers pay only when invoking a paid Actor.

Current services

ServiceActorPrimary purposeBilling eventCurrent price
AI Research Briefindustrial_platform/research-brief-agentCurrent, cited multi-source researchresearch-brief-completed$1.99
Web Change Intelligenceindustrial_platform/web-change-intelligenceDetect deterministic changes in public web contentpage-comparison$0.01
Web Metadata Intelligenceindustrial_platform/web-metadata-intelligenceExtract page metadata, OG/Twitter, JSON-LD, headingsmetadata-extraction$0.01
Article Content Intelligenceindustrial_platform/article-content-intelligenceExtract clean article text for RAG/research pipelinesarticle-extraction$0.002
PDF Text Intelligenceindustrial_platform/pdf-text-intelligenceExtract PDF text, per-page text, and metadatapdf-extraction$0.0015
Sitemap Intelligenceindustrial_platform/sitemap-intelligenceDiscover and expand sitemaps into clean URL rowssitemap-url$0.0001
Link Intelligenceindustrial_platform/link-intelligenceExtract structured page links, anchor text, rel flags, and internal/external classificationlink-extraction$0.0005

Live pricing and execution eligibility should be verified against Apify Store before paid execution.

Official MCP Registry

Industrial Platform is published in the official MCP Registry as:

text
io.github.industrial-platform-ai/industrial-platform

Registry version: 1.0.1 — publication automated on merge.

The Registry listing points to the same hosted seven-service Apify MCP bundle used by this repository.

Individual MCP Registry entries

For search-oriented discovery, each paid service also has its own active MCP Registry entry:

ServiceMCP Registry name
Research Briefio.github.industrial-platform-ai/research-brief
Web Changeio.github.industrial-platform-ai/web-change
Web Metadataio.github.industrial-platform-ai/web-metadata
Article Extractorio.github.industrial-platform-ai/article-extractor
PDF Textio.github.industrial-platform-ai/pdf-text
Sitemapio.github.industrial-platform-ai/sitemap
Link Extractorio.github.industrial-platform-ai/link-extractor

These individual entries point to the same Apify-hosted Actor tools as the portfolio bundle and improve discovery for capability-specific MCP searches.

Machine-readable discovery

The repository publishes:

text
catalog.jsonmanifests/mcp-bundle.jsonexamples/mcp/remote-server.json

Service catalog

catalog.json is the portfolio-level machine-readable catalog. It contains Actor IDs, Store URLs, billing events, current prices, and common delegation targets.

MCP bundle manifest

manifests/mcp-bundle.json defines the hosted MCP endpoint and the Actors included in the bundle.

MCP client configuration

examples/mcp/remote-server.json can be used as a starting point for MCP clients that accept remote server configuration.

Agent-to-agent paid quick start

bash
npm install -g @apify/mcpcmcpc x402 initmcpc connect "https://mcp.apify.com?tools=industrial_platform/research-brief-agent,industrial_platform/web-change-intelligence,industrial_platform/web-metadata-intelligence,industrial_platform/article-content-intelligence,industrial_platform/pdf-text-intelligence,industrial_platform/sitemap-intelligence&payment=x402" @industrial-platform --x402mcpc @industrial-platform tools-list

Fund the generated wallet with USDC on Base before paid calls. Product #7 (Link Intelligence) is not included here until its public/agentic publication is confirmed.

Quick start

MCP

Configure one remote MCP server:

json
{  "mcpServers": {    "industrial-platform": {      "url": "https://mcp.apify.com?tools=industrial_platform/research-brief-agent,industrial_platform/web-change-intelligence,industrial_platform/web-metadata-intelligence,industrial_platform/article-content-intelligence,industrial_platform/pdf-text-intelligence"    }  }}

Compatible clients can then discover the exposed Actor tools from that server.

REST / Apify API

Every service can also be invoked directly through Apify's Actor API.

General Actor identifier form:

text
industrial_platform~ACTOR-NAME

Example:

text
industrial_platform~pdf-text-intelligence

Do not commit API tokens, wallet credentials, secrets, or private keys to this repository.

When should an agent delegate?

A calling agent should prefer a specialized Industrial Platform service when delegation is cheaper, simpler, or operationally safer than rebuilding the capability internally.

Examples:

text
Need current multi-source research with citations→ research-brief-agent
text
Need to know whether a monitored page changed→ web-change-intelligence
text
Need Open Graph, JSON-LD, canonical, robots, and headings→ web-metadata-intelligence
text
Need clean article text for a RAG or research pipeline→ article-content-intelligence
text
Need text and metadata from a public PDF→ pdf-text-intelligence
text
Need structured links, anchor text, and internal/external classification→ link-intelligence

Portfolio architecture

text
developer / agent / orchestrator             ↓   Industrial Platform MCP             ↓  specialized machine service             ↓   structured result / event             ↓calling system continues workflow

Execution and monetization currently run through Apify.

Authentication and machine payments

Paid execution requires a supported authenticated or agentic-payment path.

Depending on the integration, this may use:

  • Apify OAuth
  • Apify API tokens
  • Apify-supported agentic payment infrastructure

Store eligibility and pricing can change. Calling systems should inspect current Actor metadata before executing paid services.

Repository structure

text
industrial-platform-agent-tools/│├── README.md├── catalog.json├── AUTONOMOUS_REVENUE_INITIATIVE.md│├── actors/│   ├── web-change-intelligence/│   ├── web-metadata-intelligence/│   ├── article-content-intelligence/│   ├── pdf-text-intelligence/│   ├── sitemap-intelligence/│   └── link-intelligence/│├── packages/│   ├── research-mcp/│   └── change-mcp/│├── manifests/│   └── mcp-bundle.json│├── examples/│   ├── mcp/│   ├── curl/│   ├── python/│   ├── javascript/│   ├── openai-agents/│   └── microsoft-agent-framework/│└── scripts/    └── apify/

Revenue initiative

The repository contains automation for:

  • testing and deployment
  • Apify runtime configuration
  • pay-per-event pricing
  • Store / agentic-payment eligibility
  • external paid-event telemetry
  • milestone issues at 1, 10, and 100 external paid events

See:

text
AUTONOMOUS_REVENUE_INITIATIVE.md

Security principles

Industrial Platform services should:

  • block private/local-network fetch targets when accepting arbitrary URLs
  • bound response sizes, timeouts, and concurrency
  • use structured machine-readable outputs
  • avoid silently fabricating unavailable data
  • charge successful results rather than failed fetches where practical
  • keep credentials in secret stores rather than source code

License

Integration examples and open-source code in this repository are provided under the MIT License unless otherwise noted.

Industrial Platform

Organization:

text
industrial-platform-ai

Apify namespace:

text
industrial_platform

Machine-readable catalog:

text
catalog.json

來源：README.md，提交 dcb23c4

工具

0
工具後設資料尚未被收錄。

版本歷史

1
  1. v1.0.2最新Sep 30, 2026