Vidgrid — eyes for video

vidgrid

[PyPI] [Python] [License: MIT] [Downloads]

Convert video clips into annotated image grids for vision LLM analysis. One cell = one second, by default.

[vidgrid example — a 3×3 grid generated from "Me at the zoo" with numbered cells, top-right timestamps, and real auto-captions burned in]

LLMs can't watch video, but they can analyze a single image. vidgrid samples one frame per second from a video, tiles them into a numbered storyboard with timestamps, and optionally sends the result to Claude, GPT, or Gemini with a prompt. The result is something close to "my LLM just watched a video" for the cost of a handful of image uploads.

The model

One cell = one second, by default. The auto-picker chooses the smallest grid (biggest, most-legible cells) whose board count stays under --max-boards (default 10). When that's not enough for a long clip, it bumps the grid up; as a last resort, it reduces the sampling rate. Override with --fps and --max-boards for full control.

• Grid size determines how many seconds fit in one photo
• Default sampling is 1fps; drops below 1fps only when needed to stay under the max-boards cap
• Videos over 5 minutes are rejected (chop them up first)

Quality degrades with bigger grids. Cells shrink, detail is lost, and the LLM has a harder time reading fine content like text or UI elements. Stick with 3×3 unless you specifically need to pack more seconds into one photo. 5×5 exists mostly as a "let me see what happens" option.

How many photos a video produces

At 1fps sampling, the board count at each grid size:

Bold = what auto picks — the smallest grid (biggest cells) that keeps the board count under --max-boards (default 10).

¹ At the 5-min cap, even 4×4 exceeds 10 boards at 1fps, so auto drops the sampling rate (≈1 cell per 1.9s) to land at the 10-board limit. Use --fps 1.0 --max-boards 20 to preserve 1fps and accept more boards.

Most vision LLMs accept ~10–20 images per request, so auto's default ceiling of 10 keeps a full video inside a single model call.

Install

pip install vidgrid                         # core renderer onlypip install vidgrid[transcribe]             # + faster-whisper for --transcribepip install vidgrid[anthropic]              # + Claude support via --askpip install vidgrid[llm]                    # + Claude + GPT + Geminipip install vidgrid[all]                    # everything

Requires Python 3.9+ and ffmpeg on your PATH.

Quick start

# 1. Auto-pick grid and sampling rate — smallest grid that fits in 10 boardsvidgrid clip.mp4 -o grid.png
# 2. Force a specific gridvidgrid clip.mp4 -o grid.png --grid 4x4
# 3. Force a sampling rate — 0.5fps = 1 cell every 2 secondsvidgrid long-clip.mp4 -o grid.png --fps 0.5
# 4. Raise the max-boards ceiling (default 10) if you want more boardsvidgrid lecture.mp4 -o grid.png --max-boards 20
# 5. Render + auto-transcribe + send to Claude in one callvidgrid lecture.mp4 --transcribe --ask "bullet-point summary"
# 6. Use existing Whisper captions, burn them onto the gridvidgrid interview.mp4 -o grid.png --captions whisper.json --burn-captions
# 7. Let the CLI fall back to python -m if the console script isn't on PATHpython3 -m vidgrid clip.mp4 -o grid.png

Three things you can do with it

1. Summarize a talk without watching it

vidgrid "team-meeting.mp4" \  --transcribe \  --ask "list the decisions made and who owns each" \  --model claude-opus-4-7

vidgrid samples one frame per second, runs Whisper on the audio, sends the grid + transcript to Claude, and prints the answer. The model correlates frames and words via the burned-in timestamps.

2. Find a specific moment in a screen recording

vidgrid bug-repro.mp4 --grid 3x3 \  --ask "at which numbered frame does the error dialog appear?" \  --model gpt-5

Because cells are globally numbered (1, 2, 3...) and tagged with timestamps, the model can point you at the exact moment. No scrubbing.

3. Rank a pile of stock footage

for clip in broll/*.mp4; do  vidgrid "$clip" -o "grids/$(basename $clip .mp4).png"done

Send the PNGs to Claude in a single request and ask it to rank or reject clips against your shot list. This is the workflow vidgrid was built for.

Portrait vs landscape

vidgrid keeps the grid shape square (N×N) regardless of source orientation and preserves the source aspect inside each cell. Landscape sources produce wide boards; portrait sources produce tall boards. Cells are never cropped.

Two-layer captions (default)

The default mode gives the LLM two correlated inputs: the rendered grid image AND the Whisper transcript as separate text. The model correlates them via the timestamps printed on each cell.

This beats burning captions into the image because:

1. Frames keep their pixels for actual content
2. Text is higher fidelity as tokens than as baked-in pixels
3. The grid stays clean and shareable

Add --burn-captions if you want a self-contained image (useful for sharing or offline analysis).

Caption file formats

vidgrid reads and writes three caption formats. The --captions flag auto-detects from the file extension. The --transcript-format flag controls what --transcribe writes.

JSON (default, Remotion-compatible):

[  {"text": "hello", "startMs": 0, "endMs": 500, "timestampMs": 0, "confidence": 0.98},  ...]

SRT (SubRip subtitles):

100:00:00,000 --> 00:00:00,500hello
200:00:00,500 --> 00:00:01,000world

TXT (plain timestamped text, one word per line):

0.00 hello0.50 world

Use any format as input, output, or both. You can mix — read an .srt and write a .txt with --captions foo.srt --transcript-format txt.

Python API

from vidgrid import render
storyboard = render(    input_path="interview.mp4",    output_path="grid.png",    grid="3x3",  # or "2x2", "4x4", "5x5", or None for auto    transcribe=True,)
print(storyboard.board_paths)        # ['grid-1.png', 'grid-2.png', ...]print(storyboard.transcript_path)    # 'grid-transcript.json'print(storyboard.all_samples)        # list[Sample] with timestamps

Modules: vidgrid.probe, vidgrid.sample, vidgrid.compose, vidgrid.captions, vidgrid.llm, vidgrid.presets.

Output structure

Single-board run:

grid.png                 # the storyboardgrid.json                # sidecar: timestamps, layout, source infogrid-transcript.json     # only if --transcribe or --captions was used

Multi-board run:

grid-1.png, grid-2.png, grid-3.png, ...grid.json                # index covering all boards + global cell numberinggrid-transcript.json

Cells are numbered globally across boards. A 3-board run has cells 1–27 so the LLM can reference any frame without ambiguity.

Limits and caveats

• 5-minute hard cap on video length. Longer videos are rejected. Chop them up with ffmpeg -ss START -t 300 input.mp4 chunk.mp4.
• No scene detection. v1 samples strictly 1 frame per second, uniform. No dedupe, no shifting — the spacing is always exactly 1 second.
• Variable-framerate videos may have sub-frame seek drift (≤1 frame), which is acceptable at 1fps sampling.
• Bigger grids hurt legibility. A 5×5 grid has cells ~300px wide; fine for people and objects, marginal for dense text or code. Stick with 3×3.
• LLM integration uses the official SDKs (anthropic, openai, google-genai) and won't be installed unless you request them as extras.

Prior art

• IG-VLM — research paper proving the grid trick works
• llm-video-frames — Simon Willison's per-frame approach
• vcsi — contact sheets without LLMs
• byjlw/video-analyzer — whisper + sequential frames

vidgrid's differentiator: 1 cell = 1 second, numbered cells, simple CLI, multi-provider LLM integration in one package.

License

MIT. The bundled Source Sans 3 font is licensed under SIL OFL 1.1.