Scrape.ly

scrapely-mcp

An MCP server for Scrape.ly. It gives AI agents and MCP clients access to a Scrape.ly account: search the Store for Actors, read their README and input schema, run them, follow the runs, and read the results from datasets, key-value stores and logs. It can also manage saved tasks and search the Scrape.ly and Crawlee documentation.

Full documentation: scrape.ly/docs/mcp.

The same package runs in two ways:

• Hosted: scrapely-mcp-http, a multi-tenant streamable HTTP server. Scrape.ly runs it behind the API at https://mcp.scrape.ly. Most users connect to that URL and never install anything.
• Local: scrapely-mcp-server, a stdio server started by a desktop MCP client on your machine.

Connect to the hosted server

Point your client at https://mcp.scrape.ly.

• Clients that support MCP authorization find the OAuth flow on their own and ask you to sign in.
• Otherwise send your API token as a header: Authorization: Bearer <token>.
• Clients that cannot set headers can append ?token=<token> to the URL. URLs tend to end up in logs, so prefer the header.

Example for a client that takes a JSON config:

{  "mcpServers": {    "scrapely": {      "url": "https://mcp.scrape.ly",      "headers": { "Authorization": "Bearer YOUR_TOKEN" }    }  }}

Run locally (stdio)

Requires Python 3.11 or newer.

uvx --from scrapely-mcp scrapely-mcp-server# orpip install scrapely-mcpscrapely-mcp-server

The token comes from SCRAPELY_TOKEN, or from the token field of ~/.scrapely/credentials.json or ~/.scrapely/auth.json. A token is required unless every selected tool works without one (search-actors, fetch-actor-details and the docs tools).

{  "mcpServers": {    "scrapely": {      "command": "uvx",      "args": ["--from", "scrapely-mcp", "scrapely-mcp-server"],      "env": { "SCRAPELY_TOKEN": "YOUR_TOKEN" }    }  }}

Options:

Tools

Default categories: actors, docs, runs, storage, tasks, automations. The dev category (report-problem) is off by default.

search-scrapely-docs searches the Scrape.ly docs at scrape.ly/docs by default (docSource: "scrapely"), or the Crawlee docs (crawlee-js, crawlee-py). fetch-scrapely-docs returns a page as Markdown; it accepts pages under https://scrape.ly/docs and https://crawlee.dev.

Whenever call-actor, get-actor-run or an Actor tool is enabled, get-actor-run, get-actor-output, get-dataset-items, get-key-value-store-record and abort-actor-run are added as well.

call-actor waits up to waitSecs (0 to 45, default 30) and returns the run status and storage IDs; the agent then fetches the results with get-actor-output (pass the runId, or a datasetId). It returns a page of dataset items (offset, limit up to 1000, default 100, and fields with dot notation) plus the run's OUTPUT record. Long text values are cut and a page may hold fewer items than limit so the reply stays small; hasMore and nextOffset say where to continue. get-dataset-items remains available for finer control. Clients that support protocol tasks can run call-actor as a long-running task and collect the result when the run ends. Progress notifications are sent while a run is awaited, and cancelling the request aborts the run.

Any Scrape.ly API GET endpoint can also be read with resources/read using its full URL, for example https://api.scrape.ly/v2/datasets/<id>/items?limit=20.

Hosted server

scrapely-mcp-http --host 0.0.0.0 --port 8000

URL parameters on the MCP endpoint: tools, actors, maxActorMemoryBytes, ui, payment (skyfire or x402), telemetry-enabled, token.

Each session gets its own server instance bound to the SHA-256 fingerprint of the credential that opened it; a request with a different credential is answered as if the session did not exist. SCRAPELY_TOKEN is never used by the HTTP server.

A container image definition is in Dockerfile (docker build --build-arg SOURCE=local -t scrapely-mcp .).

Configuration

Telemetry is off unless it is enabled and a write key is configured; the package ships no keys. Logs never contain tokens, payment tokens or payment signatures.

Session log and the context parameter

Every tool accepts an optional context string: one or two sentences from the AI on what the user wants and why it calls the tool. It is removed before the arguments are validated and never reaches the tool or the Actor; clients that do not send it work as before.

The server reports each session's activity to the Scrape.ly API with your own token, so Scrape.ly can see how the MCP server is used and what fails: the client name and version, the transport, each tool call with its arguments, the context, the status, the duration and a short summary of the result. Values are sent and stored as they are, nothing is masked, so do not put secrets in tool arguments you do not want Scrape.ly to see. Set SCRAPELY_MCP_SESSION_LOG=false to turn it off.

Widgets

MCP Apps widget tools (search-actors-widget, fetch-actor-details-widget, call-actor-widget, get-actor-run-widget) are implemented but disabled by default, and no widget HTML is bundled. With widgets disabled every client gets the plain tool set.

Development

uv venv && uv pip install -e ".[test]"pytest

License

MIT, see LICENSE. Third-party notices are in NOTICE.