CanYouGrab API
Domain availability lookup API with subscription billing, built on FastAPI + DNS (Unbound resolver).
Live services:
- API:
https://api.canyougrab.it
- Developer portal:
https://portal.canyougrab.it
- Auth:
https://auth.canyougrab.it (Auth0 custom domain)
Architecture Overview
Directory Structure
API Endpoints
Public API (API key auth: Authorization: Bearer cyg_...)
Portal API (Auth0 JWT auth)
Internal / Webhook
Core Request Flow
Domain Availability Check
Billing / Subscription Flow
Subscription Plans
Authentication
API Key Auth (public API consumers)
- Format:
Authorization: Bearer cyg_<token>
- Keys are SHA-256 hashed in the
api_keys table (plaintext never stored)
- Key prefix (
cyg_XXXXXXXX...) is stored for display in the portal
- Full key returned only once at creation time
- Soft-delete on revocation (
revoked_at timestamp, not physically deleted)
JWT Auth (portal/dashboard)
- Auth0 tenant:
dev-mqe5tavp6dr62e7u
- Custom domain:
auth.canyougrab.it
- Algorithm: RS256 with JWKS validation (1-hour cache)
- Audience:
https://api.canyougrab.it
- Social logins: Google, Apple (Sign in with Apple)
Domain Availability via DNS
Domain availability is checked by querying a dedicated Unbound recursive DNS resolver running on a separate DigitalOcean droplet. The worker sends NS record queries over the VPC private network and interprets the response:
The Unbound resolver caches aggressively (7 days for registered domains, 5 minutes for NXDOMAIN) and queries TLD authoritative servers directly, avoiding public resolver rate limits.
Database Schema
PostgreSQL is used for authentication, usage tracking, and billing — not for domain lookups.
PostgreSQL Tables
Valkey (Redis) Data Structures
Infrastructure
Servers
External Services
Deployment Pipelines
Production Deploy
Trigger: Push a git tag matching v* (e.g., v1.0.5)
Pipeline (.github/workflows/deploy.yml):
- GitHub Actions triggers on
v* tag push
- SSH into production server (
DEPLOY_HOST secret) using DEPLOY_SSH_KEY
- Bootstraps the target ref on the server (
git fetch + git checkout <tag>)
- Runs the repo-managed deploy script:
/opt/canyougrab-repo/scripts/deploy-host.sh
Dev Deploy
Trigger: Push to the dev branch
Pipeline (.github/workflows/deploy-dev.yml):
- GitHub Actions triggers on push to
dev
- SSH into dev server (
DEV_DEPLOY_HOST secret) using same DEPLOY_SSH_KEY
- Bootstraps the
dev ref on the server (git fetch + git checkout dev)
- Runs the repo-managed deploy script:
/opt/canyougrab-repo/scripts/deploy-host.sh
Repo-Managed Host Deploy Script
Both pipelines use a small inline SSH bootstrap to update /opt/canyougrab-repo to the target ref, then call scripts/deploy-host.sh from that checked-out revision. The repo-managed deploy script handles:
pip install -r requirements.txt for backend dependencies
rsync of backend, portal, and MCP source trees into their runtime directories
- Reinstall or refresh the
mcp-server package/runtime when the host also runs canyougrab-mcp.service
- Restart of FastAPI (uvicorn), worker, and MCP services via systemd
If /mcp is served by the same host as the API, backend-only deploys are not enough. The MCP service must be updated and restarted during the same deploy or the OAuth metadata and live MCP behavior can drift apart.
An existing /opt/deploy.sh can still be kept as a manual compatibility shim if desired, but the automated pipeline should treat the repo copy of scripts/deploy-host.sh as the source of truth.
Branching Strategy
Environment Variables
Backend (required on servers)
GitHub Actions Secrets
Local Development
Backend
Requires DNS_RESOLVER_HOST pointing to an Unbound instance (or use 8.8.8.8 for basic testing), plus PostgreSQL and Valkey/Redis (local or tunneled to managed instances).
Portal
Or use the configured launch server:
Portal dev server hardcodes API_BASE to https://api.canyougrab.it in portal/docs/src/config.ts. To develop against a local backend, change this to http://localhost:8000.
SEO: .it TLD Geo-Targeting Requirements
Because canyougrab.it uses a .it country-code TLD (Italy), all user-facing HTML (portal, docs) must include signals that the content targets English speakers:
<html lang="en"> on every page.
- Hreflang tags in
<head>:
- Canonical tag — Self-referencing
<link rel="canonical" href="{page_url}" />.
- All content in English — No Italian text on any page.
This applies to the portal (portal.canyougrab.it), API docs, and any other publicly rendered HTML.
Key Design Decisions
- Live DNS lookups: Domain availability is checked via NS queries to a dedicated Unbound recursive resolver. This gives real-time results (no 24-hour zone file lag), works for any TLD automatically, and avoids the operational complexity of daily batch zone file loading. Unbound caches aggressively (7 days for registered domains) so repeated queries are ~1ms.
- Nullable availability: DNS has more failure modes than a database lookup. When Unbound returns SERVFAIL or times out, the API returns
available: null instead of a potentially dangerous false positive. Consumers should treat null as "could not determine."
- Job queue for bulk checks: The bulk endpoint uses RQ (Redis Queue) backed by Valkey to dispatch jobs with automatic retries (2 retries, 5s/30s backoff) and failed-job tracking. Workers parallelize DNS queries across a thread pool.
- Long-polling: The bulk check endpoint holds the HTTP connection open (polling Valkey every 0.3s, up to 45s) rather than requiring clients to implement polling. Job results are stored in custom Valkey hashes (not RQ's built-in result storage) so the poll logic is independent of the queue framework.
- API keys with SHA-256 hashing: Keys are hashed before storage (like passwords), so a database breach doesn't expose raw keys.
- Stripe metadata linking: Stripe customers are linked to Auth0 users via
auth0_sub metadata on the Stripe customer object, avoiding a separate mapping table.
- Usage double-tracking: Both daily (
usage_log_daily) and per-minute (usage_log_minute) usage are recorded for monthly quota and per-minute rate limiting respectively.
- No Zuplo gateway policies: The project originally used Zuplo as an API gateway but migrated to direct FastAPI (commit
3abef70). The Zuplo portal/docs framework is still used for the developer portal.
Processes on Servers
Core services (every host)
- FastAPI (uvicorn): Serves all HTTP endpoints on port 8000
- RQ Worker (worker.py): Processes domain check jobs from the RQ queue
- MCP server (if
/mcp is hosted on this server): Serves remote MCP clients
Monitoring stack (API host, optional)
- Prometheus (localhost:9090): Scrapes metrics, evaluates alert rules
- Alertmanager (localhost:9093): Routes alerts to Slack
- redis_exporter (localhost:9121): Exports Valkey metrics to Prometheus
- RQ metrics exporter (localhost:9122): Exports queue depth, worker count, failed jobs
- Grafana (localhost:3000, nginx-proxied): Dashboards for queue health
- Auto-scaler (optional): Scales worker droplets up/down via DO API
Install the monitoring stack with: sudo bash scripts/setup-monitoring.sh
All active services on the host must be managed via systemd and restarted during deployments.
All automated deploys should route through scripts/deploy-host.sh, which is responsible for restarting whichever of these services are installed on the host.