Web Crawler MCP by InsightfulPipe

com.insightfulpipePublished by insightfulpipe.comv1.0.0Updated Nov 4, 2026

Hosted Web Crawler MCP server for Claude, ChatGPT and Cursor, with each action switched per account.

VerifiedStreamable HTTPWeb executableWeb Search & Scraping Data & Analytics

Server details

Identifier: com.insightfulpipe/crawler
Transport: Streamable HTTP
Runtime: Web executable
Trust: Verified
Published: Oct 21, 2026

Security note

MCP servers receive tool arguments from a conversation and may perform external actions. Review what a server can do before you enable it, and give it only the access it needs.

The registry verified that this server's namespace belongs to insightfulpipe.com, but the endpoint is not on that domain, so the publisher may not run this server.

SourceWeft's checks cover whether the listing is complete and the endpoint answered; they do not review the server's code or what its tools do.

Add a badge to your README

Show readers this MCP server is listed on SourceWeft. The badge links to this page.

[![Listed on SourceWeft](https://sourceweft.com/badge/mcp/com.insightfulpipe%2Fcrawler.svg)](https://sourceweft.com/mcp/com.insightfulpipe%2Fcrawler?utm_source=readme&utm_medium=badge)

Overview

AI-generated overview

Hosted SEO crawler MCP server that lets an AI assistant audit websites, extract links and brand details, and generate sitemaps and robots.txt.

What you can do: Run 25+ SEO checks on a URL, including a full technical and content audit, indexability, canonical, headings, meta tags, structured data and hreflang
Check performance and delivery: page speed, page load, mobile responsiveness and viewport, image optimization, status codes and redirect chains
Inspect security and AI-crawler access: SSL certificate, security headers, robots.txt directives, llms.txt and AI bot permissions
Extract links, categorize internal and external links, and map internal linking opportunities
Extract brand design elements such as colors, fonts, buttons and logos
Generate XML sitemaps and robots.txt content
When to use it: Useful for SEO professionals, developers and agencies who want to run site audits, technical SEO checks, link analysis or competitor assessments conversationally instead of through a separate crawler tool.
Requirements: A remote MCP endpoint added as a custom connector or HTTP server; no local package. An InsightfulPipe account with OAuth sign-in is required, and the server is included in a paid InsightfulPipe plan starting at $29.99/month after a 7-day trial. ChatGPT access depends on plan and developer mode.

Before you install

Requires signing in to an InsightfulPipe account and a paid subscription. The server crawls the URLs you give it, so target sites and their content are fetched by a third party. Actions are read-only by default and can be toggled per account; a read-only connection cannot enable write actions.

Installation

In SourceWeft

Open Web Crawler MCP by InsightfulPipe in the dashboard and add it to a workspace.
Connect the required credentials. They are stored privately and never shown on public pages.
Enable the server for the chats that should use its tools.

Web executable via Streamable HTTP. Remote servers run from the web runtime once configured in a workspace.

Use with an agent

Paste this prompt into Claude Code, Codex, Cursor or another agent — or into SourceWeft chat. The agent reads this server's install guide, shows you its publisher, endpoint and sign-in, and adds it to your client once you agree. Keys never go into a command.

Read https://sourceweft.com/mcp/com.insightfulpipe%2Fcrawler/install.md and connect the MCP server it describes to my client. Before changing any config, show me its publisher, endpoint and sign-in method and wait for my OK. Never put an API key in a command or in a file I would commit; ask me where it should come from.

Read the install guide the agent follows

Other MCP clients

Add this to your client's mcpServers config. The client completes OAuth sign-in on first connect.

{
  "mcpServers": {
    "web-crawler-by-insightfulpipe": {
      "type": "http",
      "url": "https://crawler.insightfulmcp.com/"
    }
  }
}

README

Web Crawler MCP Server (SEO Crawler) by Insightful Pipe

[MCP Compatible] [Insightful Pipe] [License]

Comprehensive SEO auditing and web crawling through AI with MCP.

Part of the Insightful Pipe MCP Server Collection — The Web Crawler MCP server enables Claude, ChatGPT, Cursor, and other AI assistants to perform SEO audits, extract data, and analyze websites. Run 25+ specialized SEO checks, generate sitemaps, and extract brand information.

[Explore All MCP Servers]

[Web Crawler MCP Server]

MCP Server URL

What is Web Crawler MCP?

Web Crawler MCP is a remote Model Context Protocol server that provides AI assistants with comprehensive SEO auditing and web crawling capabilities. This integration allows you to:

Run 25+ specialized SEO checkers on any URL
Generate XML sitemaps and robots.txt files
Extract brand design elements from websites
Audit technical SEO, content quality, and security
Analyze mobile responsiveness and page speed

Installation

Claude

Copy the MCP Server URL: https://crawler.insightfulmcp.com/
Open Claude Connectors Settings
Scroll to the bottom and click Add custom connector
Paste the URL and click Add
Click Connect and sign in to InsightfulPipe if asked. Claude then lists the connector as connected.

ChatGPT

Custom MCP servers are added through ChatGPT's Developer mode. Availability depends on your ChatGPT plan, and workspace admins may need to allow it.

Turn on Developer mode in ChatGPT settings
Create a new app for a remote MCP server and paste the URL: https://crawler.insightfulmcp.com/
Sign in to InsightfulPipe if asked. The connection finishes as soon as you are signed in.

See OpenAI's guide: Developer mode and MCP apps in ChatGPT

Claude Code

Cursor

Add the server to ~/.cursor/mcp.json (all projects) or .cursor/mcp.json (one project):

When Cursor shows Needs authentication, click Connect and sign in to InsightfulPipe if asked.

Available Actions

33 actions: 33 read, 0 write.

Read Actions (33)

Show all 33 read actions

Action	Description
`ai-crawler-checker`	Check permissions for AI training bots and crawlers
`amp-checker`	Verify AMP implementation quality and discoverability
`brand-extractor`	Extract brand design details from a website (colors, fonts, buttons, logos)
`canonical-checker`	Validate canonical URLs and conflicting hints
`character-encoding-checker`	Validate declared charset and encoding consistency
`content-optimizer-checker`	Analyze on-page content quality and keyword density
`content-quality-checker`	Score readability, word count, and duplication risk
`deprecated-html-checker`	Detect obsolete tags or attributes still in use
`favicon-checker`	Confirm favicon/icon discovery across devices
`headings-checker`	Review heading hierarchy (H1-H6) for structure gaps
`hreflang-checker`	Review international hreflang annotations
`image-optimization-checker`	Check alt text, formats, and compression levels
`indexability-checker`	Review robots/meta directives affecting indexation
`internal-linking-checker`	Map internal links and surface opportunities
`language-checker`	Verify html[lang] attributes and language targeting
`link-analysis-checker`	Detect broken or low-quality links on page
`link-extractor`	Extract all links from a page and categorize them (internal and external)
`llms-txt-checker`	Parse llms.txt instructions for AI crawlers
`meta-tags-checker`	Inspect meta title/description tags across the page
`mobile-responsive-checker`	Audit responsive layout rules and breakpoints
`mobile-viewport-checker`	Ensure viewport meta tags and responsive meta configs are in place
`page-load-checker`	Measure page weight and render-blocking assets
`page-speed-checker`	Measure load times and highlight slow resources
`robots-txt-checker`	Parse robots.txt directives and highlight blocking issues
`robots-txt-generator`	Generate robots.txt content with optional AI bot restrictions
`security-headers-checker`	Check CSP/HSTS and other security headers
`seo-audit-checker`	Comprehensive SEO audit covering technical and content issues
`sitemap-generator`	Crawl the target domain and produce an XML sitemap
`social-media-checker`	Validate Open Graph and Twitter Card metadata
`ssl-certificate-checker`	Inspect certificate chain, expiry, and HTTPS enforcement
`status-code-checker`	Trace HTTP status/redirect chains for a URL
`structured-data-checker`	Validate Schema.org markup and structured data coverage
`url-structure-checker`	Evaluate keyword usage and cleanliness of URLs

Control What Your AI Can Do

You decide what AI agents can do with each connected account:

Turn individual actions on or off for every connected account, so agents only see the actions you allow.
Connect as Read-only or Read & Write. A read-only connection can only enable read actions.
Destructive actions stay off by default. Actions such as deletes are disabled until an admin enables them.
Team access per account. Restricted team members only use the accounts they are granted, with the read actions enabled on them.

Usage Examples

Full SEO Audit

Check Technical SEO

Analyze Internal Links

Generate Sitemap

Extract Brand Elements

Security Audit

Why Web Crawler MCP?

For SEO Professionals

Comprehensive audits - 25+ specialized checks
Technical SEO - Robots, canonical, structured data
Content analysis - Quality scoring and optimization

For Web Developers

Debug issues - Find broken links, redirect chains
Performance - Page speed and resource analysis
Security - SSL and header validation

For Agencies

Client audits - Quick site assessments
Competitive analysis - Analyze competitor sites
Reporting - Generate findings conversationally

Security & Privacy

Secure requests - HTTPS support

Pricing

The Web Crawler MCP server is included in every InsightfulPipe plan, together with all other MCP servers and the CLI. Plans start at $29.99/month, and you can try it for 7 days. See insightfulpipe.com/pricing for current plans.

Ready-Made Skills and Prompts

Claude skills for SEO and marketing — ready-made skills that run on your connected data
Comprehensive SEO Site Audit
Technical SEO Health Check
Content Quality Assessment

Explore More MCP Servers by Insightful Pipe

Visit insightfulpipe.com/mcp-servers to discover our full collection of MCP servers for marketing and analytics.

SEO & Performance MCP Servers

PageSpeed MCP - Performance analysis
Google Search Console MCP - Search analytics
Screenshot MCP - Visual capture
Google My Business MCP - Local SEO

Analytics MCP Servers

Google Analytics MCP - Website analytics
Enrichment MCP - Data enrichment

View All MCP Servers →

Resources

Support

Documentation: insightfulpipe.com/docs
All MCP Servers: insightfulpipe.com/mcp-servers
Email: [email protected]

Insightful Pipe — AI-powered marketing analytics through MCP servers. Explore all integrations →

License

MIT License - see LICENSE for details.

Source: README.md at commit 7a050e9

Tools

This server asks you to sign in, so SourceWeft could not read its tools without an account and has not indexed them. What a connected account sees can depend on that account and its permissions.

Version history

v1.0.0LatestOct 21, 2026

Score

85% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

Reached in the last 30 daysRequired25 pts
Auth declaredRequired10 pts
Tools listed15 pts
Answer tools/list without a credential, or ship a package that starts in a sandbox without one.
DescriptionRequired15 pts
Logo10 pts
Documentation15 pts
License10 pts