Sitemap URL Extractor for AI Agents

io.github.industrial-platform-aiv1.0.2Updated Sep 30, 2026

x402 sitemap discovery and URL extraction for agents. $0.0001 per emitted URL.

VerifiedStreamable HTTPWeb executableWeb Search & ScrapingData & Analytics

Overview

AI-generated overview

Lets an assistant discover and expand public sitemaps into clean URL rows, billed per emitted URL.

What it does
This hosted MCP endpoint exposes the Sitemap Intelligence service, which discovers sitemaps and expands them into clean URL rows. It is one of several paid Industrial Platform services reachable through the same Apify-hosted endpoint, where tool discovery is free and callers pay only when a paid Actor runs. Results are returned as structured machine-readable output.
When to use it
Use it when an assistant needs a list of URLs from a site's sitemap for crawling, indexing, or research pipelines, and delegating that extraction is simpler than building it. It fits agent workflows that already call other Industrial Platform services.
Requirements
A remote MCP client pointed at the hosted Apify endpoint; no local package is needed. Paid execution requires a supported authenticated or agentic-payment path, such as Apify OAuth, an Apify API token, or x402 payment with a funded wallet. Network access to the endpoint is required.
Before you install
Calls are paid per emitted URL, so cost scales with the number of URLs returned; verify current pricing and eligibility before executing. Paid execution needs credentials or wallet funds, and the README warns not to commit API tokens, wallet credentials, secrets, or private keys. The service fetches arbitrary public URLs, so review its handling of private or local-network targets.

Installation

In SourceWeft

  1. Open Sitemap URL Extractor for AI Agents in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Web executable via Streamable HTTP. Remote servers run from the web runtime once configured in a workspace.

Other MCP clients

Add this to your client's mcpServers config.

{
  "mcpServers": {
    "sitemap": {
      "type": "http",
      "url": "https://mcp.apify.com?tools=industrial_platform/sitemap-intelligence&payment=x402"
    }
  }
}

README

Industrial Platform Agent Tools

Open-source integration toolkit for connecting AI agents, MCP clients, autonomous workflows, agent orchestrators, and developer applications to Industrial Platform machine-callable services.

Industrial Platform provides specialized paid utilities that other agents can discover and invoke instead of rebuilding research, monitoring, extraction, and document-processing capabilities inside every workflow.

One MCP endpoint for the full portfolio

text
https://mcp.apify.com?tools=industrial_platform/research-brief-agent,industrial_platform/web-change-intelligence,industrial_platform/web-metadata-intelligence,industrial_platform/article-content-intelligence,industrial_platform/pdf-text-intelligence,industrial_platform/sitemap-intelligence&payment=x402

This hosted MCP endpoint exposes the six currently runnable Industrial Platform services through one direct x402 integration. Tool discovery is free; autonomous callers pay only when invoking a paid Actor.

Current services

ServiceActorPrimary purposeBilling eventCurrent price
AI Research Briefindustrial_platform/research-brief-agentCurrent, cited multi-source researchresearch-brief-completed$1.99
Web Change Intelligenceindustrial_platform/web-change-intelligenceDetect deterministic changes in public web contentpage-comparison$0.01
Web Metadata Intelligenceindustrial_platform/web-metadata-intelligenceExtract page metadata, OG/Twitter, JSON-LD, headingsmetadata-extraction$0.01
Article Content Intelligenceindustrial_platform/article-content-intelligenceExtract clean article text for RAG/research pipelinesarticle-extraction$0.002
PDF Text Intelligenceindustrial_platform/pdf-text-intelligenceExtract PDF text, per-page text, and metadatapdf-extraction$0.0015
Sitemap Intelligenceindustrial_platform/sitemap-intelligenceDiscover and expand sitemaps into clean URL rowssitemap-url$0.0001
Link Intelligenceindustrial_platform/link-intelligenceExtract structured page links, anchor text, rel flags, and internal/external classificationlink-extraction$0.0005

Live pricing and execution eligibility should be verified against Apify Store before paid execution.

Official MCP Registry

Industrial Platform is published in the official MCP Registry as:

text
io.github.industrial-platform-ai/industrial-platform

Registry version: 1.0.1 — publication automated on merge.

The Registry listing points to the same hosted seven-service Apify MCP bundle used by this repository.

Individual MCP Registry entries

For search-oriented discovery, each paid service also has its own active MCP Registry entry:

ServiceMCP Registry name
Research Briefio.github.industrial-platform-ai/research-brief
Web Changeio.github.industrial-platform-ai/web-change
Web Metadataio.github.industrial-platform-ai/web-metadata
Article Extractorio.github.industrial-platform-ai/article-extractor
PDF Textio.github.industrial-platform-ai/pdf-text
Sitemapio.github.industrial-platform-ai/sitemap
Link Extractorio.github.industrial-platform-ai/link-extractor

These individual entries point to the same Apify-hosted Actor tools as the portfolio bundle and improve discovery for capability-specific MCP searches.

Machine-readable discovery

The repository publishes:

text
catalog.jsonmanifests/mcp-bundle.jsonexamples/mcp/remote-server.json

Service catalog

catalog.json is the portfolio-level machine-readable catalog. It contains Actor IDs, Store URLs, billing events, current prices, and common delegation targets.

MCP bundle manifest

manifests/mcp-bundle.json defines the hosted MCP endpoint and the Actors included in the bundle.

MCP client configuration

examples/mcp/remote-server.json can be used as a starting point for MCP clients that accept remote server configuration.

Agent-to-agent paid quick start

bash
npm install -g @apify/mcpcmcpc x402 initmcpc connect "https://mcp.apify.com?tools=industrial_platform/research-brief-agent,industrial_platform/web-change-intelligence,industrial_platform/web-metadata-intelligence,industrial_platform/article-content-intelligence,industrial_platform/pdf-text-intelligence,industrial_platform/sitemap-intelligence&payment=x402" @industrial-platform --x402mcpc @industrial-platform tools-list

Fund the generated wallet with USDC on Base before paid calls. Product #7 (Link Intelligence) is not included here until its public/agentic publication is confirmed.

Quick start

MCP

Configure one remote MCP server:

json
{  "mcpServers": {    "industrial-platform": {      "url": "https://mcp.apify.com?tools=industrial_platform/research-brief-agent,industrial_platform/web-change-intelligence,industrial_platform/web-metadata-intelligence,industrial_platform/article-content-intelligence,industrial_platform/pdf-text-intelligence"    }  }}

Compatible clients can then discover the exposed Actor tools from that server.

REST / Apify API

Every service can also be invoked directly through Apify's Actor API.

General Actor identifier form:

text
industrial_platform~ACTOR-NAME

Example:

text
industrial_platform~pdf-text-intelligence

Do not commit API tokens, wallet credentials, secrets, or private keys to this repository.

When should an agent delegate?

A calling agent should prefer a specialized Industrial Platform service when delegation is cheaper, simpler, or operationally safer than rebuilding the capability internally.

Examples:

text
Need current multi-source research with citations→ research-brief-agent
text
Need to know whether a monitored page changed→ web-change-intelligence
text
Need Open Graph, JSON-LD, canonical, robots, and headings→ web-metadata-intelligence
text
Need clean article text for a RAG or research pipeline→ article-content-intelligence
text
Need text and metadata from a public PDF→ pdf-text-intelligence
text
Need structured links, anchor text, and internal/external classification→ link-intelligence

Portfolio architecture

text
developer / agent / orchestrator             ↓   Industrial Platform MCP             ↓  specialized machine service             ↓   structured result / event             ↓calling system continues workflow

Execution and monetization currently run through Apify.

Authentication and machine payments

Paid execution requires a supported authenticated or agentic-payment path.

Depending on the integration, this may use:

  • Apify OAuth
  • Apify API tokens
  • Apify-supported agentic payment infrastructure

Store eligibility and pricing can change. Calling systems should inspect current Actor metadata before executing paid services.

Repository structure

text
industrial-platform-agent-tools/│├── README.md├── catalog.json├── AUTONOMOUS_REVENUE_INITIATIVE.md│├── actors/│   ├── web-change-intelligence/│   ├── web-metadata-intelligence/│   ├── article-content-intelligence/│   ├── pdf-text-intelligence/│   ├── sitemap-intelligence/│   └── link-intelligence/│├── packages/│   ├── research-mcp/│   └── change-mcp/│├── manifests/│   └── mcp-bundle.json│├── examples/│   ├── mcp/│   ├── curl/│   ├── python/│   ├── javascript/│   ├── openai-agents/│   └── microsoft-agent-framework/│└── scripts/    └── apify/

Revenue initiative

The repository contains automation for:

  • testing and deployment
  • Apify runtime configuration
  • pay-per-event pricing
  • Store / agentic-payment eligibility
  • external paid-event telemetry
  • milestone issues at 1, 10, and 100 external paid events

See:

text
AUTONOMOUS_REVENUE_INITIATIVE.md

Security principles

Industrial Platform services should:

  • block private/local-network fetch targets when accepting arbitrary URLs
  • bound response sizes, timeouts, and concurrency
  • use structured machine-readable outputs
  • avoid silently fabricating unavailable data
  • charge successful results rather than failed fetches where practical
  • keep credentials in secret stores rather than source code

License

Integration examples and open-source code in this repository are provided under the MIT License unless otherwise noted.

Industrial Platform

Organization:

text
industrial-platform-ai

Apify namespace:

text
industrial_platform

Machine-readable catalog:

text
catalog.json

Source: README.md at commit dcb23c4

Tools

0
Tool metadata has not been indexed yet.

Version history

1
  1. v1.0.2LatestSep 30, 2026