refd

refd

[CI] [License: MIT] [Try refd.ai] [Cloudflare Workers]

Open-source AI search monitoring — track how AI answers talk about any brand: visibility, mentions, citations, and rank across ChatGPT, Perplexity, Gemini, Google AI Mode, and Google AI Overviews. Use the hosted app at refd.ai or self-host the whole stack.

A Bun-workspace monorepo of three independently deployed Cloudflare Workers: apps/api (api.refd.ai) — the Hono API, OAuth, remote MCP, daily cron, and queue consumer, holding every binding; apps/dashboard (dash.refd.ai) — the React SPA as an assets-only Worker; apps/web (refd.ai) — the static Astro public site. Shared runtime-neutral code lives in packages/core. Data via BrightData (dataset scrapers + SERP API), stored in D1 (Drizzle) with gzipped raw payloads in R2.

refd also exposes the same workspace intelligence to AI agents through a remote MCP connector. Its default data:read scope is read-only; the optional data:write scope adds the bounded setup lifecycle, can start one onboarding report, and can revoke the connection with explicit confirmation. The hosted guide lives at refd.ai/agents; the MCP connector guide covers Claude, Claude Code, ChatGPT, generic clients, personal access tokens for headless agents, and self-hosting setup.

One click from the guide installs it in Cursor or VS Code — or claude mcp add-json refd '{"type":"http","url":"https://api.refd.ai/mcp"}'. Headless agents use a personal access token from Settings → Personal access tokens instead of OAuth.

How it works

Accounts hold workspaces; each workspace tracks one brand — its competitors, its prompt set, its runs. A daily cron creates an idempotent run for every workspace eligible for scheduled monitoring, then fans out queue messages: one batch-snapshot trigger per dataset surface × sample (trigger → notify → fetch, with a backstop poll), plus one sync SERP call per prompt × sample for AI Overviews. Every answer is scored for every tracked entity (mentioned / cited / first-mention position), and the raw payload is archived gzipped in R2. A missing AI Overview is recorded as a valid "no AIO shown", not a failure. Run dispatch persists immutable launch plans and resumes with bounded backoff. Storage converges under queue redelivery, but BrightData has no idempotency key, so a crash after provider acceptance can repeat paid work.

Develop

bun installbun run dev    # applies local migrations, then runs all three Workers behind               # Caddy on same-site subdomains over real HTTPS

Register on the login screen (business email + password ≥ 8 chars) — your first workspace is created automatically and drops you into a resumable setup wizard: name your brand, let it draft your description from your site, pick competitors and prompts, choose engines, then watch the first report fill in live. Every AI step falls back to typing it yourself. Standard accounts can create up to five workspaces, keep up to 25 active prompts in each, and enable up to three AI surfaces. Emails in ADMIN_EMAILS have no workspace or active-prompt cap and can enable all available surfaces.

bun run dev starts Caddy (Caddyfile) fronting all three Workers on same-site subdomains — Astro at https://refdlocal.io, the dashboard at https://dash.refdlocal.io, the API at https://api.refdlocal.io (add all three to /etc/hosts as 127.0.0.1, and run caddy trust once). Real HTTPS on same-site subdomains exercises secure cross-origin cookies, CORS, OAuth redirects, and WebSocket streaming just like production. Caddy stops when dev exits.

Local secrets go in apps/api/.dev.vars (gitignored): JWT_SECRET, BRIGHTDATA_API_TOKEN, BRIGHTDATA_WEBHOOK_SECRET (optional locally; requires a publicly reachable PUBLIC_BASE_URL), ADMIN_EMAILS (comma-separated administrator and operator allowlist), and EXA_API_KEY (onboarding's competitor search, optional). Non-secret build-time origins (VITE_API_ORIGIN, PUBLIC_DASHBOARD_ORIGIN, …) are committed per environment in each app's .env.development / .env.production. Analytics is a self-hosted Umami instance shared by the public site and the dashboard, so the activation funnel can span both: the web app needs PUBLIC_UMAMI_HOST_URL, PUBLIC_UMAMI_WEBSITE_ID, and PUBLIC_ANALYTICS_HOSTNAME, and the dashboard needs the same three as VITE_* (with the same website id). Leave any of them unset and the build emits no tracker at all, which is what a self-hosted deployment wants. Onboarding also uses two bindings, not secrets — Workers AI (AI) and Browser Rendering (BROWSER) — both remote: true in apps/api/wrangler.jsonc, so local dev proxies to the real services and the account needs both enabled.

• bun run check — typecheck (all workspaces) · bun run lint / lint:fix — Biome · bun test — unit tests
• bun run build — build all three Workers · bun run deploy — build, migrate D1, then deploy all three (the self-hosted/manual path; on refd.ai, merging to main deploys all three Workers automatically via Cloudflare)

Git worktrees are supported out of the box with wt: wt create <name> runs the checked-in .wt/setup.sh, which installs dependencies and copies local secrets and wrangler state, so a fresh worktree behaves like your main checkout.

Self-host

wrangler queues create refd-ingest && wrangler queues create refd-ingest-dlqwrangler secret put JWT_SECRETwrangler secret put BRIGHTDATA_API_TOKENwrangler secret put ADMIN_EMAILS       # comma-separated admin/operator allowlistwrangler secret put EXA_API_KEY        # optional: onboarding competitor searchbun run db:migrate:remotebun run deploy

bun run deploy builds all three Workers and deploys them (refd-api, refd-dashboard, refd-web); point their custom domains at api., dash., and the apex. The API workspace's own deploy script runs remote D1 migrations first, so deploying @refd/api alone (CI or a Cloudflare dash deploy command) always brings the schema along. (The hosted deployment wires these deploy scripts to push-to-main via Cloudflare's Git integration, so refd.ai itself has no manual deploy step.) You'll need a Cloudflare account (Workers paid plan for Queues, plus Workers AI and Browser Rendering enabled for onboarding), a D1 database + R2 bucket (ids/names in apps/api/wrangler.jsonc), and a BrightData account: fill the dataset IDs in apps/api/wrangler.jsonc vars (dashboard → Web Scrapers → each AI scraper) and create a SERP API zone matching BRIGHTDATA_SERP_ZONE. Set the deployment origins (PUBLIC_BASE_URL, DASHBOARD_ORIGIN, PUBLIC_SITE_ORIGIN, API_ORIGIN) and SCHEDULED_MONITORING_POLICY=all for a self-hosted deployment; the checked-in entitled policy is for refd.ai and limits cron to active pilot or subscribed workspaces. Cron schedule (daily 06:00 UTC), samples, and geo also live in apps/api/wrangler.jsonc.

Notes

• AI answers are non-deterministic: hosted scheduled runs use SAMPLES=1 per prompt and surface; read trends across completed runs, not one sample. Operators can request additional samples for focused manual runs.
• Each standard full run is capped at 25 active prompts × 3 enabled surfaces × samples (25 × 3 × 1 = 75 records at the default). Quota scales with the number of workspaces eligible for scheduled monitoring.
• The public site is built to be read by agents as well as people: every page is also served as markdown at the same path with .md appended, /llms.txt follows the llmstxt.org format and links those markdown versions, and /llms-full.txt inlines them all. Self-hosted deployments get this for free.
• Design system: DESIGN.md. Scoring/metrics contract: docs/METRICS.md. Remote MCP + OAuth: docs/mcp.md.

Contributing

Contributions welcome. See CONTRIBUTING.md for setup and the checks your PR must pass, and CODE_OF_CONDUCT.md for community expectations. Found a security issue? Follow SECURITY.md instead of opening a public issue.

License

MIT © refd