io.github.dunfred由 GitHub 账号 dunfred 发布v0.1.5更新于 Oct 30, 2026
Read videos like images: index once locally, query the text layer, fetch frames on demand.
在本地把视频索引成文字层加关键帧，让助手无需反复重新读取视频就能阅读、检索并查看画面。
Desktop only，通过 STDIO。 STDIO 服务会启动本地进程，因此需要 SourceWeft 桌面宿主。
参照 仓库 中的启动说明。
Index a video once. Query it cheaply, forever, with any LLM.
videoctx ("video context") turns any video into an open, portable Video
Context index (.vctx, see SPEC.md) using only local, model-free
tools: ffmpeg keyframes, PySceneDetect shot boundaries, faster-whisper
transcripts, and optional one-line visual captions from any cheap VLM. An
agent answers most questions from the index's text layer for fractions of a
cent, and pulls individual frames on demand only when a question needs pixels.
Ask questions about videos the way you'd ask about a document:
Who this is for: developers and anyone comfortable running a few terminal commands. You'll need Python, ffmpeg, and an Anthropic API key for question-answering (indexing itself is free and offline). If that's not you yet, star/watch the repo: a no-install version is on the roadmap.
Native video ingestion re-tokenizes the whole video on every question: an hour of footage is ~250-350k tokens, every single time. Proprietary video platforms index into closed representations behind per-minute pricing. videoctx is the third option: an open on-disk format any model can consume, produced by free tools, where the marginal cost of a follow-up question on an hour-long video is about $0.02.
Full protocol and honest cost accounting (including prompt-cache write premiums) in paper/draft.md. Query model: Claude Opus 4.8.
Numbers below are single-touch runs of the frozen configs (v0.2: captions + OCR + evidence rules, Opus 4.8 loop; v0.3, 2026-10-21: adds text-evidence rules, verbatim transcript quotes and a final letter-vs-evidence check) under full cost accounting. Where we measured the same configuration more than once, we report the observed range: on n=150, single runs wobble by 2-3 questions; on n=30, by 2-3 questions (= 7-10 points). Treat sub-3-point deltas on this benchmark as noise.
Subset composition caveat: videos were taken in Video-MME dataset order, which clusters by domain. The medium and long cells are entirely Knowledge-domain videos; the short cells are 76-88% Knowledge (remainder Film & Television). Read these as Knowledge-dominated subsets of Video-MME, not the full benchmark; future splits will be domain-stratified.
Short videos, held-out test half (n=150):
Medium videos, 4-15 min (n=30, wide error bars):
Medium is our weakest cell and remains the named target: naive leads on raw accuracy. Ten questions failed identically across five configs (prompt rules, a text-view bug fix, dense captions, a 24-thumb contact sheet); the sixth config, a 60-thumbnail 512px contact sheet baked into the cached prefix, recovered one. The receipts tell a consistent story: these videos are fast-cut and fully captioned, and the remaining gap is visual detail bandwidth at question time, naive's one structural advantage. We publish the ladder rather than only its best rung.
Correction (2026-10-21): a text-view bug had been dropping the header line (duration, shot count) on every OCR-enriched index since OCR shipped; all v0.2/v0.3 numbers were measured under that bug. v0.4 is the corrected configuration (short +2.7 points, long back to 80.0%, medium unchanged).
Long videos, 30-60 minutes (n=30):
Cost correction (2026-10-22): the v0.5 long run started 57 minutes after the v0.4 run, and for the 7 of 10 videos whose text views were unchanged by dense captioning, it silently inherited v0.4's still-live 1-hour prompt caches, flattering first-question costs. $0.109/q is the cold-equivalent (first question $0.285, follow-ups $0.021 as measured). Benchmark runs are now cache-salted per config so this cannot recur. | Uniform frame sampling (60-frame cap) | 70.0% | $0.133 | $0.133 | | Gemini Flash-Lite native | 76.7% | $0.022 | $0.022 (5-9 min latency/q) | | Native ingestion (published Pro rates, est.) | pending | ~$0.50 | ~$0.50 |
One-time index cost: ~$0.10-0.25 per hour of video (captions + local OCR + local embeddings), pennies otherwise.
Cost correction history: early short-video figures ($0.0067/q) undercounted prompt-cache write premiums; the metering was fixed 2026-10-18 and all figures above are fully loaded. Details in the paper's Section 7.3.
Requires ffmpeg on PATH. videoctx index and the MCP server's core tools
run fully locally with no API key. videoctx ask uses the Anthropic API
(ANTHROPIC_API_KEY or an ant auth login profile).
The calling agent's own model reads the text layer and frames directly; the
server makes no model calls and needs no key. Tools: video_overview (cheap
text layer), survey_video (one thumbnail per shot), get_frames(path, t0, t1, detail), transcript_search, find_moments (semantic visual search,
needs videoctx embed once per index), index_video. The server's instructions
carry the evidence rule: visual answers must cite fetched frames, never
general knowledge.
eval/methods.py documents each baseline. Cost accounting counts cache
writes at their real premium (1.25x / 2x by TTL); see the paper's Section 7.3
for the accounting traps this avoids.
The .vctx directory is plain JSON + JPEG: manifest.json (provenance,
parameters, source hash), index.json (shots, keyframes, transcript,
captions), frames/. Consumers must ignore unknown keys; enrichment layers
(OCR, embeddings) are reserved in SPEC.md v0.1.
Working and benchmarked: indexing, query loop, MCP server, eval harness. Next: medium-duration subset, Pro-tier native baseline, cross-model portability runs (same indexes via non-Claude models), OCR and per-shot embedding layers, spec v1.0.
Apache-2.0. Copyright 2026 Fred Dunyo.
来源：README.md，提交 72fac17
Index a video file for querying (shots, keyframes, speech transcript). Runs locally; takes roughly 10-20% of the video's duration. Idempotent - returns immediately if the video is already indexed. The other tools auto-index too, so calling this first is only needed to control when the wait happens.
path必填The cheap text layer of a video: shot list with timestamps and visual captions (when present), interleaved with the speech transcript. Read this before fetching any frames: it answers most questions by itself.
path必填One small thumbnail per shot across the whole video (subsampled to at most 24), each labeled with its timestamp. Cheap visual sweep for locating the relevant moment before zooming in with get_frames.
path必填Frames evenly spaced across [start_s, end_s]. Frames are expensive: use the narrowest range the question allows, and raise max_frames (up to 8) only when a sparse sample missed the moment. Set detail='high' (2x resolution, ~4x token cost) when small text, colors, patterns, or fine object details matter.
path必填end_s必填detailstart_s必填max_framesSemantic visual search: find the moments that best match a natural language description (e.g. 'a red car at night'), using local CLIP embeddings. Returns the top matches as timestamped thumbnails: use the timestamps to zoom in with get_frames. Requires embeddings built once with `videoctx embed <index>`; cheaper and more targeted than survey_video when you know what you are looking for.
path必填top_kdescription必填Case-insensitive search over the transcript. Returns matching segments with surrounding context and timestamps: use the timestamps to target get_frames.
path必填query必填context_s已满足适用检查项的 90%
该评分衡量此收录条目的完整程度，依据注册中心条目、端点与公开来源计算得出。它不是对服务器本身的评价，也不表示服务器安全或当前可用：可连通性是带日期的检查结果，评分也不包含安装量或用户评价。
请在注册中心条目中声明凭据请求头，或保持端点能够响应探测，以便判定其认证方式。
videoctx index lecture.mp4 # one-time, local, freevideoctx show lecture.vctx # the text view the model readsvideoctx ask lecture.vctx "What diagram is shown when she explains backprop?"git clone <repo> && cd videoctxuv venv && uv pip install -e .claude mcp add videoctx -- $(pwd)/.venv/bin/videoctx-mcpuv pip install -e ".[eval]"python eval/prepare_videomme.py --duration short --videos 100 --out eval/datapython eval/split_dev_test.py eval/data/videomme_short.jsonlpython eval/run.py eval/data/videomme_short_test.jsonl --methods videoctx,naive