Llmstxt Doc Search

llmstxt-doc-search

Live, ranked search across any number of llms.txt documentation sites - Strands, Kiro, the AWS guides, and whatever you add at runtime.

[npm version] [MCP Registry] [License: MIT] [Release]

llmstxt-doc-search is a Model Context Protocol (MCP) server that turns the llms.txt index a documentation site publishes into a fast, ranked search tool your agent can call. It indexes titles at startup, ranks queries with BM25, and fetches the full document only when you open a result - so you get current docs with almost no local storage. Built on the search engine from @praveenc/mcp-docs-server, generalized to a runtime registry of sources.

It implements the MCP 2026-07-28 specification over stdio and still works with clients on the 2025 protocol, which open with an initialize handshake. Requires Node.js 20 or later.

Why

An llms.txt file is a curated index of a doc site's pages, published for tools like this one to consume. They can be large - AWS Bedrock's lists roughly a thousand documents - so downloading everything is wasteful and goes stale fast.

This server takes a leaner approach:

• Title-only index, built lazily. On first search of a source, only the page titles are indexed. That is fast to build and tiny to hold in memory.
• Ranked with BM25. Queries are scored with BM25 plus Porter stemming, bigrams, and markdown-aware weighting (headers, code, and links count for more). Technical terms like mcp, json, and stdio are preserved rather than stemmed.
• Content on demand. The full markdown or HTML of a result is fetched only when you call fetch_doc.

The result is a good fit for broad, fast-moving reference material - the opposite tradeoff to snapshotting docs into a local vault.

Installation

Quick start (recommended)

Add the server to your MCP client configuration (Claude Desktop, Kiro, and others). It is downloaded and run on demand via npx - no manual build:

{  "mcpServers": {    "llmstxt-doc-search": {      "command": "npx",      "args": ["-y", "@praveenc/llmstxt-doc-search"]    }  }}

Global install

npm install -g @praveenc/llmstxt-doc-search

Then point your MCP client at the installed binary:

{  "mcpServers": {    "llmstxt-doc-search": {      "command": "llmstxt-doc-search"    }  }}

Quick start

Once the server is connected, the typical flow is three calls:

1. docs_home() - orient yourself: see the registered sources and how to search and fetch.
2. search_docs("prompt caching", "aws-bedrock-userguide") - rank matching docs. Omit the source to search everything.
3. fetch_doc(url) - read the full content of a result you like.

Add your own source at any time and it is indexed immediately and persisted for future runs:

add_doc_source("langgraph", "https://langchain-ai.github.io/langgraph/llms.txt")

Tools

docs_home, list_doc_sources, search_docs, and fetch_doc are annotated read-only, so a client can approve them without prompting. add_doc_source, remove_doc_source, and refresh_doc_source change the persisted registry, and remove_doc_source is annotated destructive. The tool list is fixed, so it is advertised as cacheable for one hour.

Default sources

Seeded into the registry on first run:

strands, kiro, aws-bedrock-userguide, aws-agentic-ai-lens, aws-bedrock-agentcore-devguide, mcp.

The registry is persisted at ~/.config/llmstxt-doc-search/sources.json (override with LLMSTXT_REGISTRY_PATH). Anything you add, remove, or refresh at runtime is saved there.

Configuration

All configuration is via environment variables; none are required.

Testing with MCP Inspector

npx @modelcontextprotocol/inspector npx -y @praveenc/llmstxt-doc-search

The Inspector can connect in either protocol era; see Protocol eras.

Development

Clone the repository for local work (Node.js 20 or later):

git clone https://github.com/praveenc/llmstxt-doc-search.gitcd llmstxt-doc-searchnpm install

Commands

npm run dev         # run from source with tsx (no build)npm test            # offline unit and protocol testsnpm run typecheck   # type-check without emittingnpm run build       # compile to dist/npm run inspect:dev # MCP Inspector against the source

Local MCP client config (development)

Point your client at a source checkout instead of the published package:

{  "mcpServers": {    "llmstxt-doc-search": {      "command": "npx",      "args": ["tsx", "/ABS/PATH/llmstxt-doc-search/src/index.ts"]    }  }}

Or, after npm run build, at the compiled entry point:

{  "mcpServers": {    "llmstxt-doc-search": {      "command": "node",      "args": ["/ABS/PATH/llmstxt-doc-search/dist/index.js"]    }  }}

Architecture

src/├── index.ts              # Tool registration and the stdio entry point (serves 2026-07-28 and 2025-era clients)├── config.ts             # Defaults and environment configuration├── tools/│   └── docs.ts           # search_docs, fetch_doc, and source management└── utils/    ├── doc-fetcher.ts    # HTTP fetching, redirect handling, HTML parsing    ├── indexer.ts        # BM25 search index    ├── registry.ts       # Persisted source registry    ├── store.ts          # In-memory document store    ├── text-processor.ts # Tokenization and snippet helpers    ├── url-validator.ts   # SSRF guard and URL validation    ├── stopwords.ts      # Stop-word list    └── logger.ts         # Logging utilities

Search algorithm

Ranking uses BM25 (Best Matching 25) with several enhancements:

• Porter stemming matches word variants (for example, running and run).
• Bigrams capture phrase matches (for example, prompt caching).
• Weighted scoring boosts title matches (3-8x), headers (4x), code blocks (2x), and link text (2x).
• Domain-term preservation keeps technical terms like mcp, json, and stdio unstemmed so they match exactly.

Security

This server fetches user-supplied URLs at runtime, so its SSRF surface is guarded in depth:

• Scoped fetches. fetch_doc only retrieves URLs that a registered source's llms.txt lists exactly, or that sit under that source's origin and path prefix (matched on a path boundary rather than a raw string prefix). There is no arbitrary fetch.
• Scheme allow-list. Non-http(s) schemes are rejected.
• Range-based address blocking. Private and reserved destinations are blocked using IP range classification (ipaddr.js), covering decimal, octal, and hex IPv4, IPv4-mapped IPv6, loopback, link-local, unique-local, carrier-grade NAT, and other reserved ranges - not just a hostname regex.
• Connection-time validation. The resolved IP is checked at connection time via a custom DNS lookup, closing DNS-rebinding, and every redirect hop is re-validated.
• Bounded responses. Response bodies are capped at 10 MB to limit memory and regular-expression (ReDoS) exposure.

Runtime dependencies report zero known vulnerabilities.

License

MIT - Copyright (c) 2026 Praveen Chamarthi

Contributing

Contributions are welcome. If you find a bug or have an idea:

1. Open an issue describing the problem or proposal.
2. For code changes, fork the repo and create a feature branch.
3. Keep changes focused, add or update tests, and make sure npm test, npm run typecheck, and npm run build all pass.
4. Open a pull request against main with a clear description of what changed and why.

Commit messages follow the Conventional Commits style.

Support

• Questions and ideas: open a GitHub issue.
• Bugs: please include your MCP client, the tool call you made, and any relevant logs (set LLMSTXT_LOG_LEVEL=debug for more detail).
• Security issues: open an issue marked as security-sensitive, or contact the maintainer directly rather than posting exploit details publicly.