Clickcast

🎞️ clickcast — Playwright browser automation, screenshots, and GIF/MP4/WebP recording for AI agents

Give AI agents visual + structured feedback about live web UIs — and give humans deterministic demo reels while you're at it.

[PyPI version] [Python] [CI] [License: MIT]

What's new (v0.4.4) — npx-installable npm wrappers (clickcast + clickcast-mcp) on top of the v0.3.0 highlights: live agent control via an MCP server, pixel-level clickcast diff, accessibility semantics fused with the pixel-grid overlay, an official CI GitHub Action, Homebrew/apt packaging, and a self-healing first run. See CHANGELOG.md for the full notes.

[clickcast · scripted tour of tailwindcss.com — sticky A→B arrow, actions panel bottom-right, symmetric open/close pairs]

See docs/ONE_PAGE_NAVIGATION_ORDER_TIPS.md for the nine principles behind why this reel reads as legibly as it does — and the scenario template you can copy for your own reels.

clickcast drives a real browser through a website and hands back two things:

1. A watchable reel — GIF / MP4 / WebP / raw frames.
2. A machine-readable JSON sidecar — every step's selector, timings, per-step frame paths, discovered elements, and post-action page state (title, URL, console errors, failed requests). Versioned. See docs/feedback-schema.md.

Point it at a URL and it will auto-discover the interactive elements and build a tour for you, or hand it a small YAML scenario for a scripted, repeatable walkthrough.

Install

One line, ready to run immediately after:

pip install clickcast && clickcast install --with-deps chromium && clickcast doctor

Broken down — pip install clickcast (requires Python ≥ 3.10) gets you the CLI; clickcast install downloads Chromium (~one-time, ~180MB — kept out of the pip package itself since it's versioned independently and every project doesn't need every engine); --with-deps also pulls the system libraries Chromium needs (Linux only, may prompt for sudo); clickcast doctor confirms everything above actually worked.

Forgot the second step? Any command that needs a browser (auto, run, shot, elements, mcp) detects a missing engine itself and offers to install it right then instead of failing — say yes once and it retries automatically. That prompt only fires in an interactive terminal; CI/scripted runs fail fast with the exact fix command instead of hanging on stdin.

Homebrew (macOS/Linux)

brew install --build-from-source ./Formula/clickcast.rb   # works today, clone this repo firstbrew install AlexKay28/clickcast/clickcast                # tap not bootstrapped yet -- see docs/packaging/homebrew.md

apt (Debian/Ubuntu, .deb)

bash scripts/build_deb.sh 0.2.9 && sudo dpkg -i dist-deb/clickcast_0.2.9_amd64.deb   # works today, clone this repo firstsudo apt install clickcast                                                           # hosted repo not bootstrapped yet -- see docs/packaging/apt.md

Both native packages skip the Chromium download (~180MB, versioned independently of clickcast) and never install a second ffmpeg (clickcast already bundles one via imageio[ffmpeg]) -- run clickcast install --with-deps chromium once after either install path. Full design rationale, what's live today vs. what needs one-time bootstrapping, and the exact bootstrap steps: docs/packaging/homebrew.md, docs/packaging/apt.md.

npm (clickcast + clickcast-mcp)

Two npm packages -- thin Node wrappers whose postinstall provisions an isolated Python venv and pip-installs the pinned PyPI clickcast, since there's no way to ship clickcast's Playwright/Pillow/ffmpeg runtime as pure JS. Built specifically because the MCP ecosystem's install pattern is npx <package>, not pip install:

npx -y clickcast-mcp                 # MCP entry point -- execs `clickcast mcp`npx clickcast --version              # general CLI wrapper

Both are published on the npm registry. To run against your working copy instead of the published version:

cd npm/clickcast-mcp && npm pack && npm install -g ./clickcast-mcp-*.tgz && clickcast-mcp --helpcd npm/clickcast      && npm pack && npm install -g ./clickcast-*.tgz      && clickcast --version

Full design rationale (including why the shared provisioning code is a vendored copy rather than a file: dependency), what's live today vs. what needs bootstrapping, and the exact bootstrap steps: docs/packaging/npm.md.

For AI agents — copy this prompt into your chat

Paste the block below into your coding agent (Claude Code, Cursor, Copilot Chat, Codex, etc.) to teach it clickcast in one message. The agent will install the tool, verify the environment, generate a visual + machine-readable report for your project, and know how to gate CI on the results.

You are going to use `clickcast` (https://pypi.org/project/clickcast/) to visually verifythis project's frontend. It produces a watchable GIF reel + a JSON sidecar withper-step selectors, timings, page state, error codes, and a graph of the app'spages. Follow this exact sequence:
1. INSTALL   pip install clickcast   clickcast install --with-deps chromium   # first time only, ~180 MB   clickcast doctor                          # bail out if this fails
2. LOAD FULL CAPABILITIES   Run `clickcast skill --json` and read the output. It lists every subcommand,   flag, docs URL, and sidecar schema pointer. Prefer this over guessing.
3. RUN A QUICK TOUR OF THE APP   If we have a running URL (dev server / staging / production):     clickcast auto <URL> --for-humans --emit-events --out tour.gif   If we have a specific flow to verify, write a YAML scenario and run:     clickcast run tour.yml --emit-events --out tour.gif   `--for-humans` gives a legible reel for the user to watch;   `--emit-events` prints a machine-readable JSONL line you can parse.
4. READ THE SIDECAR (JSON at `<gif>.json`, schema v4)   For each step, gate on the structured fields — NOT regex over prose:     - `status`      : "ok" | "failed" | "skipped"     - `error_code`  : "timeout" | "locator_missing" | "cross_origin" |                       "navigation_error" | "selector_ambiguous" | "other"     - `skip_reason` : "optional_no_reaction" | "pre_action_failed" |                       "element_vanished" | "cross_origin_bounce"     - `page_state`  : title, url_after, console_errors, page_errors,                       network_failed   The top-level `graph` block gives page nodes + navigation edges you can   use to reason about the app's shape, not just the sequence you ran.
5. WATCH STDERR FOR ADVISORIES   clickcast prints `⚠ <message> — see <docs-url>` lines for known anti-   patterns (nav-heavy tour, click without DOM reaction, very short reel,   cross-origin bounce, incoherent cursor styling). Each has a stable   kebab-case id you can dedupe or gate on.
6. FOR CI REGRESSION GATES   `clickcast assertions <sidecar>.json --baseline golden.json` diffs the   run against a committed baseline; nonzero exit on drift. Byte-identical   across runs (timestamps, frame paths, and URL query strings excluded).
Reference docs (all in-repo, load lazily as needed):  - Sidecar shape:   https://github.com/AlexKay28/clickcast/blob/main/docs/feedback-schema.md  - Human-legible reel authoring: https://github.com/AlexKay28/clickcast/blob/main/docs/ONE_PAGE_NAVIGATION_ORDER_TIPS.md  - Agent integration:  https://github.com/AlexKay28/clickcast/blob/main/docs/ai-integration.md
If something is unclear, run `clickcast <subcommand> --help` before asking me.

Once the agent has this, ask it something concrete like "run clickcast auto against http://localhost:3000 and tell me which clicks had DOM reactions" — it now has everything it needs.

Live agent control (MCP)

Everything above is batch mode: record a whole tour, then read back a GIF + sidecar. clickcast mcp is the live counterpart — an MCP server that drives one action at a time (goto/click/type/scroll/...) and hands back clickcast's richer per-call payload (annotated frame, page_state, grid coordinates, an enumerated error_code) instead of a bare screenshot, so an agent can react before deciding the next step.

pip install 'clickcast[mcp]'

{  "mcpServers": {    "clickcast": { "command": "clickcast", "args": ["mcp"] }  }}

Reach for mcp when an agent needs to explore and react live; reach for auto/run for a repeatable, one-shot artifact (CI, docs, release notes). Full tool reference, client config for Claude Code / Claude Desktop, and the schema doc: docs/mcp.md · docs/mcp-tool-schema.md.

First run — 30 seconds

clickcast auto https://example.com --out tour.gif

Produces two files:

• tour.gif — the reel
• tour.gif.json — the AI-consumable sidecar (schema_version: 4, spec at docs/feedback-schema.md)

For a walkthrough of how an LLM agent consumes both, see docs/ai-integration.md.

Three modes

All three are deterministic, headless-by-default, and CI-friendly.

Commands

auto <url>

Discover interactive elements and record a click-tour.

run <scenario.yml>

Execute a YAML scenario. See Scenario format.

clickcast run docs/scenarios/spa.yml \    --out release-notes.mp4 --format mp4 \    --var base_url=https://staging.example.com

Flags: --out, --format, --headful, --slowmo MS, --url URL (retarget the first goto step — see below), --var key=value (repeatable — substitute {{ key }} inside the scenario), --no-sidecar.

CLI flags override the scenario's meta: block.

Point an existing scenario at a different environment with --url — no YAML edits, no {{ URL }} templating:

clickcast run tour.yml --url https://staging.example.com/app

--url rewrites the first goto step's URL and wins over --var URL=.... Only the first goto is touched — later goto steps are usually intra-app navigation from the entry point, so they stay put.

shot <url>

Single screenshot.

clickcast shot https://example.com --full-page --out home.png

Flags: --out, --full-page, --wait (load / domcontentloaded / networkidle / a selector / a number of seconds), --viewport, --device, --engine, --dark.

init [path]

Scaffold a starter YAML scenario. --from-auto runs discovery once and seeds the file with the top-scoring click steps.

clickcast init tour.yml --url https://example.com --from-auto

Flags: --url, --name, --out, --from-auto, --force.

elements <url>

Dump the discovered interactive elements — useful for authoring selectors. Each entry additionally carries an accessibility block (Playwright's own role / accessible name / interactive state, fused with the pixel-grid overlay's grid_cell when --grid is on) — see docs/feedback-schema.md.

clickcast elements https://example.com --json > elements.jsonclickcast elements https://example.com --grid --grid-pitch 50 --json

explore <url>

Map the site's state graph and emit explore.json — auto --traversal bfs minus everything that records (no frames, no annotator, no encoder), so mapping an app costs seconds instead of a reel shoot. Output shape: docs/explore-schema.md.

clickcast explore https://example.com --out map.json

report-bug <sidecar.json>

Turn a sidecar into an actionable bug report — redacted by default, shareable as-is, or --open to launch a prefilled GitHub issue.

clickcast report-bug tour.gif.jsonclickcast report-bug tour.gif.json --json    # machine-readable Track-C payloadclickcast report-bug tour.gif.json --open --note "behind corporate proxy; TLS interception on"

Flags: --json, --open, --redact/--no-redact (default on — sanitizes URLs, selectors, and visible text), --note.

feedback

Long-form usage feedback: record a session, then summarize it.

clickcast feedback startclickcast feedback statusclickcast feedback summary --jsonclickcast feedback stop

Subcommands: start, stop, status, list, summary.

doctor

Check Python version, playwright, engine binaries, ffmpeg, config path.

clickcast doctor           # human-readableclickcast doctor --json    # machine-readable, non-zero exit on failure

config

Read / write persistent defaults.

clickcast config path                    # print the user config file pathclickcast config list                    # every effective value + sourceclickcast config get engineclickcast config set engine firefox

Set values land in the user TOML at clickcast config path. See Configuration for precedence.

install [engines…]

Wrapper over playwright install. Default engine: chromium.

clickcast install                        # chromium onlyclickcast install firefox webkit         # add moreclickcast install --with-deps chromium   # Linux: pull system libs (needs sudo)

mcp

Start a stdio MCP server for live agent control — see Live agent control (MCP) above and docs/mcp.md. Requires pip install 'clickcast[mcp]'.

clickcast mcp --grid   # every action tool's returned frame carries the coordinate overlay

Flags: --engine, --viewport, --device, --headful, --lang, --dark, --grid/--grid-pitch/--grid-color/--grid-style — all defaults for start_session when the connecting agent doesn't override them.

Scenario format

A scenario is plain YAML: a meta: block and a list of steps:. Full worked examples: docs/scenarios/.

meta:  name: WorldSight broad tour  engine: chromium              # chromium | firefox | webkit  viewport: 1280x800  device: null                  # or "iPhone 15", "Pixel 8", "iPad Pro"  fps: 12  dwell: 1.0                    # default seconds after each step  format: gif                   # gif | mp4 | webp | frames  out: worldsight.gif
steps:  - goto: https://worldsight-weld.vercel.app    wait: networkidle    label: Open WorldSight
  - click: "text=3D"    label: Switch to 3D globe    dwell: 2.0
  - hover: "[aria-label='Rankings']"  - click: "[aria-label='Rankings']"    label: Open Rankings
  - type:      into: "#search"      text: "Japan"    label: Search Japan
  - select:      in: "#metric"      value: "GDP"
  - scroll:      to: footer

Supported actions

Every step also accepts label, dwell, optional: true (don't fail the run if the selector is missing — sidecar records status: "skipped"), and repeat: N.

Variable substitution: {{ key }} inside any string, injected via --var key=value.

Python API

Fluent, chainable — every builder returns self:

from clickcast import Reel
reel_path = (    Reel("https://worldsight-weld.vercel.app", viewport=(1280, 800), fps=12)    .goto(wait="networkidle")    .click("text=3D", label="Switch to 3D globe", dwell=2.0)    .click("[aria-label='Rankings']", label="Open Rankings")    .scroll(to="footer")    .save("worldsight.gif")  # or .save("tour.mp4", quality=8))

Async variant for callers already inside a running event loop:

from clickcast import AsyncReel
reel = AsyncReel("https://example.com").goto(wait="networkidle").click("#cta")path = await reel.save("tour.gif")

Discovery only, no reel:

from clickcast import discover
elements = discover("https://example.com", limit=10)

Skip the sidecar with save(..., no_sidecar=True).

Pre-push iteration on a local static build

Building a static site and reeling it locally before pushing? Use Reel.serve_dir (or the standalone serve_directory helper) as a context manager — it starts a threaded HTTP server on a free port, yields the base URL, and tears the server down on exit. No more python3 -m http.server 8091 & invocations that leak past your shell session and collide with the next iteration.

from clickcast import Reel
with Reel.serve_dir("./public") as url:    Reel(url).goto().click(".chip").save("out.gif")# Server is gone; the port is free.

Defaults are safe for dev iteration: loopback-only bind (127.0.0.1), OS-picked free port, ThreadingHTTPServer so parallel browser requests don't queue. Override any of them explicitly:

from clickcast.serving import serve_directory  # importable without going through Reel
with serve_directory("./dist", port=8091, bind="0.0.0.0", threading=False) as url:    ...

bind="0.0.0.0" exposes the server to your LAN — opt-in, not the default.

Reading the sidecar

Every recording run writes <out>.json alongside the media file.

from clickcast.feedback import load
report = load("tour.gif.json")
for step in report.steps:    if step.status == "failed":        print(f"step {step.index} ({step.action}) failed: {step.error}")        print("  frames:", step.frames)        if step.page_state:            print("  console errors:", step.page_state.console_errors)

Consumers that don't want to import clickcast can parse the JSON directly against the schema at src/clickcast/feedback/schema/v4.json. A standalone reference implementation lives at tests/consumer/read_sidecar.py.

See docs/ai-integration.md for the two-line agent-integration example and docs/feedback-schema.md for the full field-by-field walkthrough.

CI: 2-line regression gate

On GitHub Actions? .github/actions/clickcast packages this recipe (plus the visual diff gate below, browser caching, and a PR comment with the reel + a summary table) as a reusable Action — see docs/ci/README.md. This section stays the canonical recipe for every other CI platform, and for anyone who'd rather not depend on a third-party Action for two lines of shell.

Every reel writes a JSON sidecar, but raw sidecars carry timestamps, frame filenames, and query-string tokens — none of which are stable across runs. For a proper CI regression gate, use clickcast assertions (or Reel.assertions()) to distill the sidecar down to the shape that actually matters: step count, per-step action / label / status, and the per-step error counters.

The distilled shape is byte-identical across runs of the same scenario against the same URL (schema: docs/assertions-schema/v1.json). Diff it against a committed baseline; non-zero exit on drift.

Bootstrap the baseline once:

clickcast run tour.yml --out reel.gifclickcast assertions reel.gif.json > tests/golden-tour.json  # commit this

Then in CI (2 lines):

clickcast run tour.yml --out reel.gifclickcast assertions reel.gif.json --baseline tests/golden-tour.json

Exit 0 means the target UI produced the same step ordering, statuses, and error-signal counts as when the baseline was captured; anything else is real drift and the command prints per-line descriptions like step 2: status changed 'ok' -> 'failed'.

Same signal from Python:

from clickcast import Reel
reel = Reel(url).goto().click(".cta").save("reel.gif")drift, is_clean = reel.assertions_diff("tests/golden-tour.json")if not is_clean:    raise SystemExit("\n".join(drift))

Excluded from the distilled shape on purpose: wall-clock timestamps, per-step duration_ms, frames filenames, resolved URLs (including query-string tokens), cursor_xy. If you need those in your gate too, diff the raw sidecar with your own tooling — the assertion set is the narrow "did the UI still behave" contract, not the wire-level snapshot.

Its visual companion: clickcast diff

assertions is structural — it never looks at a pixel. clickcast diff (and Reel.visual_diff()) is the pixel-level companion: it pairs up two sidecars' steps and pixel-diffs their frames, reporting a percent-changed and a list of changed bounding regions per step, plus region-highlighted diff images. Reach for assertions to catch "the flow broke" (wrong step count, a step that started failing); reach for diff to catch "the flow ran fine but the button moved / the color changed / the layout shifted." The two compose — run both in CI for full coverage, or diff alone if pixel drift is your only concern.

clickcast diff reel.gif.json tests/golden-tour.gif.json \  --out diff-report --fail-above 5

Exits non-zero when any step's changed pixel percentage exceeds --fail-above, or when a step couldn't be paired with its baseline counterpart at all (mismatched step counts fall back to label matching; anything still unmatched is flagged rather than silently skipped). --out collects the region-highlighted diff images plus a summary.json for CI artifacts; --threshold tunes the per-pixel noise floor. Clickcast's own overlays (progress bar, action label, actions panel, cursor + ripple) are excluded from the diff by default — otherwise every run would flag its own chrome as a regression — pass --no-exclude-overlays for a strict raw-pixel diff. Same signal from Python via reel.visual_diff(run_path, baseline_path).

Configuration

Precedence (highest → lowest):

1. CLI flags
2. Scenario meta: block
3. CLICKCAST_* environment variables
4. Project ./clickcast.toml
5. User TOML (path via clickcast config path)
6. Built-in defaults

Every Config field can be set at any of these layers: engine, viewport, device, headful, slowmo, lang, dark, proxy, fps, dwell, format, quality, loop.

Project TOML — flat or [defaults]-wrapped both work:

# clickcast.tomlengine   = "chromium"viewport = "1280x800"fps      = 12dwell    = 1.0format   = "gif"

Env vars:

CLICKCAST_ENGINE=firefoxCLICKCAST_VIEWPORT=1440x900CLICKCAST_HEADFUL=trueCLICKCAST_PROXY=http://proxy.internal:8080

Output formats

--quality 1..30 trades size for fidelity (lower = better). --loop 0 loops forever; --loop 1 plays once.

How it works

   URL ─▶ Session ─▶ Actions ─▶ Recorder ─▶ Encoder ─▶ .gif/.mp4/.webp        (chromium)  (auto or   (per-step   (Pillow /                     YAML)      PNGs +      imageio-                                manifest)   ffmpeg)                        │           │                        ▼           ▼              PageStateCollector    ─▶ ReportBuilder ─▶ <out>.json                (console errors,                 page errors,                 failed requests)

1. Session launches a Playwright browser at the requested viewport/device.
2. Actions run one step at a time (click, type, scroll, …) with normalised timings and cursor tracking.
3. Recorder captures a pre-frame + N padding frames per step (deterministic filenames, byte-identical copies for padding).
4. PageStateCollector subscribes to page events for the sidecar.
5. Encoder produces the final artifact; ReportBuilder finalises the JSON.

The annotator (clickcast.annotate.Annotator — click ripples, cursor trail, caption bar, progress bar) ships as a library API in v0.1. Automatic wiring into auto / run outputs is planned for v0.2 (see Roadmap).

Troubleshooting

• Blank frames — the site is a SPA; increase --initial-wait, or add wait: networkidle (or a specific selector) to the first step.
• ffmpeg not found — imageio-ffmpeg bundles a static binary; falls back if missing. Choose gif / webp if you'd rather skip MP4 entirely.
• Selector not found — clickcast elements <url> shows what's actually clickable. Or mark the step optional: true.
• Can't reach an internal site — set CLICKCAST_PROXY, or proxy in the scenario meta: block.
• Chromium missing — clickcast install. On Linux CI add --with-deps.
• Sidecar shape changed — the current schema is versioned at src/clickcast/feedback/schema/v4.json; every bump from v1 has been strictly additive (the graph block landed in v2), so older sidecars keep parsing. See docs/feedback-schema.md for each version's additive contract.

Contributing

git clone https://github.com/AlexKay28/clickcastcd clickcastpip install -e ".[dev]"clickcast install

Before opening a PR:

ruff check .ruff format --check .mypypytest -m "not integration"      # fast; ~2spytest                            # full; needs chromium

Cutting a release is documented in RELEASING.md.

Roadmap

v0.1 — Session · Actions · Recorder · Encoder · Discovery · YAML scenarios · CLI · Python API · Sidecar (schema v1) · Config precedence · Fixture test site · Docs.

v0.2 — everything on the original v0.2 list shipped during the v0.2–v0.4 line:

• On-frame HUD — shipped as the --for-humans composite (--highlight-target, --title-card, --summary-card, --zoom-on-click, onboarding pace) with the annotation pipeline wired into auto/run.
• BFS UI exploration — shipped as auto --traversal bfs (plus --seed-url for deterministic paths, visited-dedup, and the top-level graph sidecar block). No separate explore command yet — see below.
• Sidecar schema v2 with the graph block — shipped (schema is now at v4; every bump strictly additive).
• Automatic annotation of auto / run outputs — shipped (annotate.pipeline + AnnotateConfig).

v0.5 (in development on the v0.5 line):

• Flagship: first-class clickcast explore <url> — implemented (see Unreleased in the CHANGELOG): the state-graph command the v0.2 plan only gestured at. Same traversal/dedup internals as auto --traversal bfs, minus recording + encoding — returns a fast explore.json (nodes = distinct page states, edges = (from, to, action, transition_kind), plus top discoveries per state). Structured-only, agent-first — the reel is opt-in. Contract: docs/explore-schema.md.
• Pillow 14 deprecation cleanup — done: the 28 Image.getdata deprecation warnings in pixel-assertion tests migrated to a flattened_pixels helper (tests/_stubs.py) that prefers get_flattened_data and falls back across the supported Pillow>=10.0 range.
• Sidecar schema maturity — the v0.1-era plan for a graph block (originally tracked in #29) landed at schema v2 and the schema has since matured to v4 with strictly additive bumps; any future v5 keeps that contract.

Related projects

vercel-labs/webreel — a TypeScript tool for authoring polished demo videos. Easy to confuse with this project given the similar name and overlapping output, so: clickcast is a Python tool aimed primarily at AI agents that need a visual modality onto a live web UI, and secondarily at humans who want reproducible demo reels. If you want hand-authored marketing videos, webreel is the better fit.

License

MIT © 2026 Alex Kay. See LICENSE.