LLM Audit

app.llmauditPublished by llmaudit.appOfficial publisherv1.1.0Updated Nov 2, 2026

Measure whether AI assistants actually recommend a brand, from measured answers.

VerifiedStreamable HTTPWeb executableBusiness & CommerceAI & ML

Overview

AI-generated overview

Measures whether AI assistants actually recommend a brand when buyers ask for recommendations, reporting the measured answers rather than model self-reports.

What you can do
  • Start a visibility measurement for a brand domain and receive a runId
  • Poll the run until providers have answered and collect the verdict, buyer questions won, and competitors named instead
  • Read the measured map of live buyer questions and which brand won each answer
  • Consult the methodology resource explaining how the measurement works
When to use it
Useful for checking how a brand appears in AI assistant recommendations, or for tracking which competitors get named instead. Note the README states the free measurement service closed on 2026-10-29: starting a new check now returns a closed status, and only measurements started before that date can still be read.
Requirements
A remote streamable HTTP endpoint, or the npm package run locally via npx for command-only clients. No authentication is declared. Self-hosting the server involves environment variables for the API base URL, a server key, Supabase URL and service role key, and a monthly run limit.
Before you install
The README says free measurements spent the provider's API keys, roughly USD 0.05 to 0.15 each, and that the service is now closed in favor of a paid scan requiring an account. Self-hosting asks for a server key and a Supabase service role key, which are sensitive credentials. Measurements are limited to one per domain every 30 days.

Installation

In SourceWeft

  1. Open LLM Audit in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Web executable via Streamable HTTP. Remote servers run from the web runtime once configured in a workspace.

Use with an agent

Paste this prompt into Claude Code, Codex, Cursor or another agent — or into SourceWeft chat. The agent reads this server's install guide, shows you its publisher, endpoint and sign-in, and adds it to your client once you agree. Keys never go into a command.

Read https://sourceweft.com/mcp/app.llmaudit%2Fllmaudit/install.md and connect the MCP server it describes to my client. Before changing any config, show me its publisher, endpoint and sign-in method and wait for my OK. Never put an API key in a command or in a file I would commit; ask me where it should come from.

Read the install guide the agent follows

Other MCP clients

Add this to your client's mcpServers config.

{
  "mcpServers": {
    "llm-audit": {
      "type": "http",
      "url": "https://mcp.llmaudit.app/mcp"
    }
  }
}

README

askedthrice-mcp

MCP server for askedthrice.com. Lets anyone's assistant measure whether a brand shows up when buyers ask an AI for a recommendation.

Closed since 2026-10-29. Asked Thrice no longer runs free measurements; the product is a paid scan that needs an account. start_visibility_check returns status: "closed" with the link to askedthrice.com and does not call the API. get_visibility_check still reads measurements started before that date. The sections below describe how it worked while it was open.

Hosted at https://mcp.askedthrice.com/mcp (streamable HTTP). Listed in the official MCP registry as com.askedthrice/askedthrice. The stdio bridge for clients that only launch local commands is the npm package askedthrice-mcp, in npm/.

Full spec: specs/features/geo-13-mcp-server.md in the private llm-rank-tracker repo.

Install

json
{"mcpServers":{"askedthrice":{"url":"https://mcp.askedthrice.com/mcp"}}}

or, for command-only clients:

json
{"mcpServers":{"askedthrice":{"command":"npx","args":["-y","askedthrice-mcp"]}}}

What it exposes

  • start_visibility_check starts the measurement and returns a runId.
  • get_visibility_check collects it. Returns running until the providers answer.
  • askedthrice://methodology explains how it is measured.

Two tools rather than one because a measurement takes up to two minutes: the audit is 15 to 60 s and the measured map another 15 to 60 s.

What it returns, and what it does not

It returns the measured map: the buyer questions that were asked live and who won each answer. Never the providers' self-report, which is what each model believes about the brand. The two numbers drift apart a lot in practice, and the self-report is always the optimistic one; that is why the tool contract does not expose it in any field.

It never returns a 0 to 100 score: a verdict in three bands and counts.

Money

Every free measurement spends askedthrice's API keys, on the order of USD 0.05 to 0.15. Three guards, all covered by tests:

  1. One free measurement per domain every 30 days, checked BEFORE the API is called.
  2. MCP_MONTHLY_RUN_LIMIT (default 200) as the channel ceiling, separate from the web funnel budget.
  3. Rate limiting in prod counts by client identity, not by this server's IP, which rotates on serverless.

Env

VariablePurpose
LLMAUDIT_API_BASE_URLAPI base. Default https://askedthrice.com
MCP_SERVER_KEYKey shared with llm-rank-tracker. Without it prod buckets by IP
SUPABASE_URL / SUPABASE_SERVICE_ROLE_KEYmcp_runs table. Without them the store is in memory and does NOT work on serverless
MCP_MONTHLY_RUN_LIMITMonthly ceiling of runs for the channel

Channel usage

npm run stats reads mcp_runs and reports how much of the monthly ceiling is left, who called, which domains were measured and which one currently holds the 30-day quota.

sh
node --env-file <file.env> scripts/stats.mjsnode --env-file <file.env> scripts/stats.mjs --days 30 --history 180

Needs SUPABASE_URL (or NEXT_PUBLIC_SUPABASE_URL) and SUPABASE_SERVICE_ROLE_KEY. --days is the daily detail, --history how far back it looks; the current month is always included in full, because that number has to match the guard's.

The first thing it prints, when there are any, is the orphaned runs: reserved rows that never got their audit_id. They return "Unknown runId" to the client forever while holding their domain's free quota and counting against the monthly ceiling. Delete them by run_id.

Vercel Web Analytics cannot measure this: it works by injecting a script into HTML, and here there is no HTML and no browser to run it, every request is a JSON-RPC call from an MCP client. It would read zero forever.

Docker (directory checks)

Dockerfile builds the HTTP server and serves it on 0.0.0.0:3001, which is what Glama and similar directories run to verify the server starts and answers introspection. Port and host are read by xmcp.config.ts at build time, so they go as ENV before npm run build, not at docker run. Without SUPABASE_* the run store is in memory: fine for a check, not for serving traffic.

Development

sh
npm run dev     # xmcp devnpm test        # vitest, no networknpm run build   # xmcp build

Tests run against real fixtures in fixtures/ (already-paid runs from prospecting). None of them touch the network or call a provider.

Publishing a new registry version

server.json is published with mcp-publisher using DNS auth on askedthrice.com. The key lives outside the repo as a PEM; the CLI wants the raw ed25519 seed in hex:

sh
HEX=$(openssl pkey -in <key.pem> -noout -text | sed -n '/^priv:/,/^pub:/p' | grep -v '^priv:\|^pub:' | tr -d ' :\n')mcp-publisher login dns --domain askedthrice.com --private-key "$HEX"mcp-publisher publish server.json

Bump version in server.json on every publish; the registry rejects a repeat. When the npm package changes version, update packages[0].version too.

Source: README.md at commit 0c5e67c

Tools

2

get_visibility_check

Get visibility check

Collect the result of a measurement started with start_visibility_check. Returns status running while the providers are still answering, so poll every 20 to 30 seconds. When ready it reports a verdict, how many of the buyer questions the brand won, which competitors were named instead, and which providers actually answered.

ParametersrunIdrequired

start_visibility_check

Start visibility check

Measure whether a brand shows up when buyers ask AI assistants for a recommendation. Asks real buyer questions to OpenAI, Anthropic and Gemini and reports what they answered, not what they claim they would answer. Free, no signup, one measurement per domain every 30 days. Returns a runId: call get_visibility_check with it after about a minute. If the domain was already measured this month the response says quota_reached and includes when it ran.

Parametersbrandrequiredwebsiterequiredcategoryrequiredlanguagelocationcompetitors

Version history

1
  1. v1.1.0LatestSep 16, 2026

Score

100% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 daysRequired25 pts
  • Auth declaredRequired10 pts
  • Tools listedRequired15 pts
  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts