Smart Crawler

io.github.shulex-vocPublished by GitHub account shulex-vocv0.2.1Updated Oct 24, 2026

E-commerce competitor intelligence, VOC sentiment, Amazon ASIN analysis, Reddit growth playbooks.

VerifiedStreamable HTTPWeb executableWeb Search & ScrapingData & AnalyticsBusiness & Commerce

Overview

AI-generated overview

Hosted MCP server for e-commerce competitor intelligence, VOC review sentiment, Amazon ASIN analysis and Reddit growth playbooks.

What you can do
  • Search competitor product catalogs by brand, country, keyword or price
  • Pull consumer VOC reviews from Trustpilot, Reviews.io and Google Maps, plus Amazon ASIN reviews with AI pain-point analysis
  • Rank Google Shopping merchant share for a keyword
  • Generate Reddit subreddit growth playbooks for top contributors
  • Scrape a single URL with short-lived agent memory
  • Run natural-language warehouse queries
When to use it
Useful when an assistant needs market, competitor or customer-voice research for e-commerce: comparing competitor catalogs, analyzing Amazon or review-site sentiment, sizing a Google Shopping landscape, or planning Reddit community growth.
Requirements
A remote streamable-http endpoint at smartcrawler.io; no local package. Requires an API key generated in the smart-crawler console, sent as an Authorization Bearer token (sck_...). Network access to the hosted service.
Before you install
The API key is a secret sent in the Authorization header; keep it out of shared configs and logs. The service is hosted, so queries and scraped URLs go to a third party. Some tools consume credits, while the warehouse query is listed as zero-credit.

Installation

In SourceWeft

  1. Open Smart Crawler in the dashboard and add it to a workspace.
  2. Connect Authorization token credentials. They are stored privately and never shown on public pages.
  3. Enable the server for the chats that should use its tools.

Web executable via Streamable HTTP. Remote servers run from the web runtime once configured in a workspace.

Use with an agent

Paste this prompt into Claude Code, Codex, Cursor or another agent — or into SourceWeft chat. The agent reads this server's install guide, shows you its publisher, endpoint and sign-in, and adds it to your client once you agree. Keys never go into a command.

Read https://sourceweft.com/mcp/io.github.shulex-voc%2Fsmart-crawler/install.md and connect the MCP server it describes to my client. Before changing any config, show me its publisher, endpoint and sign-in method and wait for my OK. Never put an API key in a command or in a file I would commit; ask me where it should come from.

Read the install guide the agent follows

Other MCP clients

Add this to your client's mcpServers config.

{
  "mcpServers": {
    "smart-crawler": {
      "type": "http",
      "url": "https://smartcrawler.io/mcp/",
      "headers": {
        "Authorization": "Bearer <YOUR_TOKEN>"
      }
    }
  }
}

README

smart-crawler MCP

E-commerce competitor intelligence, VOC review sentiment, Amazon ASIN analysis, Google Shopping landscape, on-demand URL scraping, and Reddit subreddit growth playbooks — exposed to AI agents as a hosted Model Context Protocol server.

  • Endpoint: https://smartcrawler.io/mcp/ (streamable-http)
  • Auth: Authorization: Bearer sck_... — generate a key in the smart-crawler console
  • Registry: io.github.shulex-voc/smart-crawler

This repository hosts the public MCP manifest for the smart-crawler service. It is the discovery/connection entry point — the service itself is hosted.

Connect

Claude Code

bash
export SMARTCRAWLER_API_KEY=sck_...claude mcp add --scope user --transport http smart-crawler \  https://smartcrawler.io/mcp/ \  --header "Authorization: Bearer $SMARTCRAWLER_API_KEY"

--scope user makes the server visible in every project. Without it, claude mcp add defaults to local scope (only the current directory), a common "I added it but /mcp is empty" pitfall. Verify with claude mcp list or /mcp.

Codex (~/.codex/config.toml)

toml
[mcp_servers.smart-crawler]url = "https://smartcrawler.io/mcp/"bearer_token_env_var = "SMARTCRAWLER_API_KEY"

Any MCP client

Point your client at https://smartcrawler.io/mcp/ over streamable-http and send your API key as a Bearer token. See mcp.json for the full tool list and server.json for the registry manifest.

Tools (highlights)

ToolWhat it does
reddit_subreddit_playbookFull growth playbook for a subreddit's top contributors
search_competitor_productsSearch competitor catalog by brand / country / keyword / price
get_voc_reviewsConsumer VOC reviews from Trustpilot / Reviews.io / Google Maps
amazon_voc_reportAmazon ASIN reviews + AI VOC analysis (pain points, listing tips)
competitor_landscapeGoogle Shopping merchant-share ranking for a keyword
scrape_urlOne-URL scrape with short-lived agent memory
query_warehouseWarehouse-first natural-language query (0 credits)

Full list in mcp.json.

License

MIT

Source: README.md at commit 8e4c675

Tools

0
This server asks for a credential such as an API key, so SourceWeft could not read its tools without one and has not indexed them.

Version history

1
  1. v0.2.1LatestSep 16, 2026

Score

75% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 daysRequired25 pts
  • Auth declaredRequired10 pts
  • Tools listed15 pts

    Answer tools/list without a credential, or ship a package that starts in a sandbox without one.

  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts

    Declare a license in the repository or the registry entry.