Yueying – Let AI watch videos

yueying — let AI watch videos

Point Claude, Cursor or any MCP client at a video and get back a timestamped transcript plus keyframe contact sheets — offline, no API key. Local files first; URLs (YouTube, Bilibili, Douyin, Xiaohongshu, TikTok, Vimeo, …) are videos you are entitled to process, fetched via yt-dlp at ≤720p and deleted after processing by default.

[PyPI] [PyPI downloads] [License: MIT] [Python 3.10+] [Add to Cursor] [Install in VS Code]

中文说明 ↓

Yueying (阅影) means "read video". One package gives you an MCP server, a CLI and an agent skill.

What you get

[Claude Desktop: a YouTube link is pasted, the watch_video tool runs for about 40 seconds, and Claude answers with timestamped key points]

Claude Desktop with yueying connected: paste a link, wait about forty seconds, get the video back as timestamped notes. This video ships captions, so speech recognition never ran, and the model asked for the transcript only. Keyframes and contact sheets come back through get_frames when it needs to see the screen. Demo video: GitInGifs: Git Branches by GitLab, CC BY.

[A 3x3 contact sheet: nine keyframes, each with a yellow "#number mm:ss" label bottom-left]

Contact sheet from a 24-second demo clip (four app screenshots with Chinese narration). The yellow label on every tile is the keyframe number and timestamp; the model cites them back to you.

The transcript of the same clip — local speech recognition, language auto-detected as Chinese:

[00:00] 这是阅读,一个安静的桌面小说阅读器。整本书连续滚动,按段落记住进度。        第二个画面是桌面模式,窗口变透明,只留文字浮在桌面上。        第三个画面是伪装皮肤,一键变成代码编辑器。        最后是伪装成表格的样子。

(The app is called 月读; ASR heard the homophone 阅读. Speech recognition does that to names — the model corrects it from the on-screen text in the frames.)

Every video becomes one folder:

report.md          index for the model: metadata, chapters, contact sheets, keyframes, transcripttranscript.txt     paragraphs with [mm:ss] timestampstranscript.srt     subtitles for any playergrid_01.jpg …      3x3 contact sheets, 9 keyframes each, in time orderframes/            full-size keyframes, e.g. f003_00m15s.jpgmanifest.json      machine-readable result (paths, segments, chapters, options)

Why yueying

• Captions first, Whisper only when needed. Platform subtitles are used when they exist. Otherwise local faster-whisper: large-v3-turbo on an NVIDIA GPU, small on CPU, automatic CPU fallback — nothing is uploaded, no key.
• ffmpeg bundled. Works on Windows 11 out of the box (imageio-ffmpeg); no PATH fiddling.
• Token-efficient. Keyframes are taken at scene changes, near-duplicates dropped, then packed into 3x3 contact sheets with burned-in timestamps. One sheet ≈ 1–2K tokens for nine moments; one transcript with [mm:ss] paragraphs.
• Chinese platforms and the rest. Bilibili (multi-part, collections, member videos with your browser login), Douyin, Xiaohongshu — and YouTube, TikTok, Vimeo, X and every other yt-dlp site.
• Zero API keys, zero telemetry. The only network traffic is the video site you name and one Whisper model download. See the privacy policy.

Benchmark: a 6-minute Bilibili video → report in ~90 s on an RTX 5060 laptop; on CPU with model=small expect ~1–2 min per 10 min of speech.

Quick start

1. Install uv (Python is not required):
   bash

   winget install astral-sh.uv                         # Windowsbrew install uv                                     # macOScurl -LsSf https://astral.sh/uv/install.sh | sh     # Linux / macOS

2. Warm up and check everything once (installs the package, probes the GPU, downloads the speech model, runs a 2-second smoke test, prints config to paste):
   bash

   uvx yueying mcp --setup

3. Add the server to your client (below), then ask: "Watch C:\videos\lecture3.mp4 and turn the steps into notes" or "What does this video say about docker compose: https://www.bilibili.com/video/BV…".

Claude Desktop

%APPDATA%\Claude\claude_desktop_config.json (Windows) · ~/Library/Application Support/Claude/claude_desktop_config.json (macOS). Fully quit and reopen Claude afterwards.

{ "mcpServers": { "yueying": { "command": "uvx", "args": ["yueying", "mcp"], "env": { "PYTHONUTF8": "1" } } } }

Windows note: Claude Desktop does not always see your PATH — if the server fails to start ("spawn uvx ENOENT"), use the absolute path, e.g. "command": "C:\\Users\\<you>\\.local\\bin\\uvx.exe" (where uvx prints it). Logs: %APPDATA%\Claude\logs\mcp-server-yueying.log (~/Library/Logs/Claude/ on macOS). Keep wait_seconds at its default there; see the RUNNING rule.

Claude Code

claude mcp add --transport stdio --scope user yueying --env PYTHONUTF8=1 -- uvx yueying mcp

Or drop this repo's .mcp.json into a project (it ships with "timeout": 1800000 so one watch_video call can wait for a long video). To raise Claude Code's tool timeout globally, set MCP_TOOL_TIMEOUT=1800000 (ms) in your environment. The repo is also a Claude Code plugin (.claude-plugin/plugin.json: server + skill).

Cursor

Click the Add to Cursor badge above, or put the same JSON in ~/.cursor/mcp.json (global) or .cursor/mcp.json (project):

{ "mcpServers": { "yueying": { "command": "uvx", "args": ["yueying", "mcp"], "env": { "PYTHONUTF8": "1" } } } }

Cline

MCP Servers → Configure (cline_mcp_settings.json). timeout is in seconds; the five read-only tools are safe to auto-approve. Step-by-step agent instructions: llms-install.md.

{  "mcpServers": {    "yueying": {      "type": "stdio",      "command": "uvx",      "args": ["yueying", "mcp"],      "env": { "PYTHONUTF8": "1" },      "timeout": 1800,      "autoApprove": ["get_transcript", "search_transcript", "get_frames", "get_frame_at", "list_videos"]    }  }}

Windsurf

~/.codeium/windsurf/mcp_config.json:

{ "mcpServers": { "yueying": { "command": "uvx", "args": ["yueying", "mcp"], "env": { "PYTHONUTF8": "1" } } } }

VS Code (Copilot agent mode)

Click the Install in VS Code badge above, or create .vscode/mcp.json (note the root key servers):

{ "servers": { "yueying": { "type": "stdio", "command": "uvx", "args": ["yueying", "mcp"], "env": { "PYTHONUTF8": "1" } } } }

Direct links for hosts that accept custom URL schemes: cursor://anysphere.cursor-deeplink/mcp/install?name=yueying&config=eyJjb21tYW5kIjoidXZ4IiwiYXJncyI6WyJ5dWV5aW5nIiwibWNwIl0sImVudiI6eyJQWVRIT05VVEY4IjoiMSJ9fQ== and vscode:mcp/install?%7B%22name%22%3A%22yueying%22%2C%22command%22%3A%22uvx%22%2C%22args%22%3A%5B%22yueying%22%2C%22mcp%22%5D%2C%22env%22%3A%7B%22PYTHONUTF8%22%3A%221%22%7D%7D.

Without uv (pip / pipx) and Windows one-click

pip install yueying            # or: pipx install yueyingyueying mcp --setup            # prints a config with the absolute path of the yueying-mcp executable

Use that absolute path as "command" with no args (Windows: ...\Scripts\yueying-mcp.exe; also works as python -m yueying mcp). Windows users without Python tooling can double-click install.cmd from a checkout: it creates %LOCALAPPDATA%\yueying\venv, installs the Claude Code skill, runs yueying mcp --setup and prints the JSON block with the right path.

GPU

uvx --from "yueying[cuda]" yueying mcp     # NVIDIA: adds the CUDA runtime wheels (cuBLAS, cuDNN)pip install "yueying[cuda]"

Device and model are chosen automatically (model=auto: large-v3-turbo on CUDA, small on CPU); if the GPU trial fails, recognition falls back to CPU by itself.

Docker

docker build -t yueying .docker run --rm -i -v yueying-data:/data -v "$PWD/videos:/videos:ro" yueying

The image is CPU-only (containers get no GPU by default), so it defaults to the small model. Mount your videos read-only and give the tools container paths (/videos/lesson.mp4); results and the downloaded Whisper weights live in the /data volume. In a client config the command is docker and args are ["run", "--rm", "-i", "-v", "yueying-data:/data", "-v", "/your/videos:/videos:ro", "yueying"].

Tools

video for the read tools accepts the video_id from watch_video/list_videos, the results folder, or the same path/URL you gave watch_video.

The RUNNING rule

Processing can take minutes, and most hosts cap a tool call at about a minute. So watch_video waits at most wait_seconds, then answers RUNNING video_id=… · stage 2/4 speech recognition 40% · elapsed 46 s · est. ~1 min remaining. The agent simply calls watch_video again with the same video — it re-attaches to the same job (options are ignored while it runs; refresh=true restarts). Recommended wait_seconds:

Progress notifications are sent every 1.5 s for hosts that display them. One video is processed at a time per server; extra requests queue.

First run: the first speech recognition downloads a Whisper model once (~480 MB small on CPU, ~1.6 GB large-v3-turbo on GPU). uvx yueying mcp --setup does this ahead of time; otherwise the RUNNING line says "first run downloads ~… this can take several minutes".

Where files go

Root: $YUEYING_OUT_DIR if set, else ~/yueying_out. One entry per video, named <slug>-<video_id> (yt-<id>, bili-<BV>, or the file name) — never renamed; the title lives in manifest.json.

~/yueying_out/└── bili-BV1xx-3f9a2c1e/    ├── report.md  transcript.txt  transcript.srt  manifest.json    ├── grid_01.jpg … grid_07.jpg    ├── frames/            f001_00m02s.jpg …   (+ frames/extra/ for get_frame_at)    ├── .job               only while a job runs    └── _download/         only with YUEYING_KEEP_SOURCE=1

Disk budget: ≈ 25 MB per hour of video; 300–600 MB/h more with YUEYING_KEEP_SOURCE=1. Nothing is deleted automatically — list_videos shows sizes; delete a folder to free space; watch_video(refresh=true) reprocesses one video. Editing a local file changes its size/mtime and therefore gets a new entry.

Supported sources

• Local files: anything ffmpeg reads — mp4, mkv, mov, webm, avi, flv, ts, and audio (mp3, m4a, wav, …). Audio-only input gives a transcript without frames.
• URLs: every site yt-dlp supports. Fetched at ≤720p and deleted after processing unless YUEYING_KEEP_SOURCE=1.
• Bilibili: without login Bilibili serves 480p — enough for slides and code. For HD or member-only videos pass cookies_from_browser="chrome" (or edge, firefox, brave, chromium, safari); on Windows close Chrome first, it locks its cookie database. Multi-part videos and collections: p= links are separate entries; the CLI's --all processes them all.
• Not for live streams or images. Short links (b23.tv, v.douyin.com) are processed but not de-duplicated against their long form (the server never resolves URLs itself).

Also a CLI and an agent skill

yueying video.mp4yueying "https://www.bilibili.com/video/BVxxxx" --ui-lang enyueying "https://www.youtube.com/watch?v=xxxx" --out ./notes/xxxyueying lesson1.mp4 lesson2.mp4 "https://www.bilibili.com/video/BVyyyy"   # several at once, one folder each + index.mdyueying "https://www.bilibili.com/video/BVxxxx" --all                      # every part of a multi-part video / collectionyueying --install-skill                                                    # Claude Code skill -> ~/.claude/skills/yueying

Default output: ./yueying_out/<name>/ (parent folder when several inputs). CLI log lines and report.md are Chinese by default (--ui-lang en for English); 0.3 will flip the default to English.

The skill (src/yueying/skill/SKILL.md) tells a coding agent to prefer the MCP tools when present and otherwise run the CLI and read report.md plus the contact sheets. Tools that support the Agent Skills standard can copy ~/.claude/skills/yueying/SKILL.md into their own skills folder.

Compared with similar projects (September 2026)

"–" means the project did not advertise the feature when we looked; check their READMEs, they may have moved on.

Privacy policy

yueying collects nothing and has no telemetry, analytics, crash reporting or update checks. All processing is local. The only network connections are (1) to the video site of the URL you pass, through yt-dlp, and (2) to Hugging Face (or HF_ENDPOINT) to download a Whisper model once. Outputs are stored in your folder until you delete them. The transcript and any frames you request are sent only to the model your MCP client is configured to use — that transfer is governed by your client's and provider's terms, not by yueying. Questions: GitHub issues. Full text: docs/privacy.md.

Troubleshooting / FAQ

• "No result received" in Claude Desktop. Keep wait_seconds at 45 (the agent then re-calls watch_video), and run uvx yueying mcp --setup once so the first call is not also the model download.
• spawn uvx ENOENT / server fails to start. The host cannot see your PATH: use the absolute path to uvx (where uvx / which uvx) or to yueying-mcp as "command".
• Console windows flash on Windows. Update to 0.2.0+: child processes are started without a window. If you still see them, you are running an old install (uv cache clean yueying).
• Mojibake / ????? in titles. Add "env": { "PYTHONUTF8": "1" } to the server config (all snippets above include it).
• Bilibili error 412 / "-352". The site wants a login: cookies_from_browser="edge" or "chrome" (close Chrome first on Windows).
• YouTube "Sign in to confirm you're not a bot". Same fix: cookies_from_browser. Also try updating yt-dlp: uv cache clean yueying or pip install -U yt-dlp.
• Slow on CPU. model="small" is already the automatic choice without an NVIDIA GPU; use mode="frames" when only the pictures matter, or mode="transcript" to skip keyframes.
• Names, numbers and code are wrong in the transcript. Expected with any ASR — the agent is told to trust on-screen text; ask it to get_frame_at the moment.
• Model download is slow or blocked (mainland China). Set HF_ENDPOINT=https://hf-mirror.com in the server env (the CLI switches to the mirror automatically when huggingface.co is unreachable).
• GPU error (CUDA / cuDNN / out of memory). model="small" or YUEYING_DEVICE=cpu; install the CUDA wheels with yueying[cuda].
• Want to reprocess with different settings. watch_video(video=…, refresh=true, …) — it kills a running job for that video, deletes the entry and starts again.

How it works

video / URL ──► yt-dlp (≤720p) + platform subtitles            ──► ffmpeg 16 kHz audio ──► faster-whisper (skipped when subtitles exist)            ──► ffmpeg scene detection ──► keyframes (near-duplicates dropped)            ──► Pillow: burn "#n mm:ss", pack 3x3 contact sheets            ──► report.md · transcript.txt · transcript.srt · manifest.json
MCP client ──stdio──► mcp_server.py ──spawns──► python -m yueying.cli <video> --json --ui-lang en                       │  parses the child's progress lines, long-polls, caches per video                       └─► store.py (cache keys / folders) · query.py (paging, search, frames)

The server process never loads yt-dlp, Whisper or CUDA itself; all heavy work runs in a child process that is killed with the server. Everything lives in src/yueying/:

Roadmap

• next — .mcpb one-click bundle for Claude Desktop, Smithery listing.
• 0.3 — forget_video / prune tools, English as the CLI default report language, --all (playlists) over MCP.

Credits

• faster-whisper, yt-dlp, imageio-ffmpeg, Pillow
• Burned-in timestamps on contact sheets follow video-vision-mcp; duration-based frame intervals follow video-analyzer-skill

License

MIT — see LICENSE.

中文说明

阅影（yueying）让 AI 看懂视频。 给 Claude Desktop、Claude Code、Cursor 等支持 MCP 的工具一个本地视频文件或视频链接（B站 / YouTube / 抖音 / 小红书 …），它就能拿到带时间戳的文字稿和关键帧九宫格：字幕优先，没有字幕就本地 faster-whisper 识别，全程离线，不上传、不要 API key。链接通过 yt-dlp 以 ≤720p 下载，处理完默认删除原视频。完整文档见上方英文部分。

安装

# 1. 装 uv（不需要先装 Python）winget install astral-sh.uv          # Windows；macOS: brew install uv# 2. 预热：安装、探测显卡、下载模型、跑 2 秒冒烟测试、打印配置uvx yueying mcp --setup

不用 uv：pip install yueying（有 NVIDIA 显卡再加 pip install "yueying[cuda]"），然后 yueying mcp --setup 会打印 yueying-mcp 的绝对路径。Windows 也可以下载仓库后双击 install.cmd，它会建独立环境、装 Claude Code 技能、跑 --setup 并打印可粘贴的配置。

配置

Claude Desktop（%APPDATA%\Claude\claude_desktop_config.json，改完完全退出再打开；Windows 下 uvx 找不到就写绝对路径，如 C:\\Users\\<你>\\.local\\bin\\uvx.exe）、Cursor（~/.cursor/mcp.json）、Windsurf（~/.codeium/windsurf/mcp_config.json）都是同一段：

{ "mcpServers": { "yueying": { "command": "uvx", "args": ["yueying", "mcp"], "env": { "PYTHONUTF8": "1" } } } }

Claude Code 一行：

claude mcp add --transport stdio --scope user yueying --env PYTHONUTF8=1 -- uvx yueying mcp

Cline 在同一段里加 "type": "stdio"、"timeout": 1800，并把 get_transcript、search_transcript、get_frames、get_frame_at、list_videos 放进 autoApprove。VS Code 的 .vscode/mcp.json 根键是 servers 并加 "type": "stdio"。

六个工具

Claude Desktop / Cursor 里 wait_seconds 保持默认 45；Claude Code / Cline 配好 timeout 后可以给到 1500，一次调用就等到结果。第一次语音识别要下载模型（CPU 约 480 MB 的 small，显卡约 1.6 GB 的 large-v3-turbo），--setup 会提前下好。

输出目录与环境变量

结果在 ~/yueying_out/<名字>-<video_id>/（YUEYING_OUT_DIR 可改），每个视频一个文件夹：report.md、transcript.txt、transcript.srt、grid_01.jpg …、frames/、manifest.json。约每小时视频 25 MB；不会自动删，删文件夹即可。常用环境变量：YUEYING_OUT_DIR（输出根目录）、YUEYING_MODEL（默认 auto：有 NVIDIA 显卡用 large-v3-turbo，否则 small）、YUEYING_DEVICE（auto/cuda/cpu）、YUEYING_LANG（服务端 report.md 语言，默认 en，中文写 zh）、YUEYING_KEEP_SOURCE=1（保留下载的原视频）、YUEYING_JOB_TIMEOUT（单个视频上限秒数，默认 7200）、PYTHONUTF8=1（Windows 防乱码）。

B站 cookie

不登录 B站 只给 480p，看 PPT 和代码够用；高清或会员视频传 cookies_from_browser="chrome"（或 edge），Windows 下先关掉 Chrome，否则读不到 cookie 数据库。遇到 412 / -352 也是同样的处理。

国内镜像

下载模型慢或被墙：在服务器配置的 env 里加 "HF_ENDPOINT": "https://hf-mirror.com"。命令行版在 huggingface.co 连不上时会自动切到镜像。

命令行用法

yueying 视频.mp4yueying "https://www.bilibili.com/video/BVxxxx"yueying "https://www.youtube.com/watch?v=xxxx" --out ./notes/xxxyueying 第1课.mp4 第2课.mp4 "https://www.bilibili.com/video/BVyyyy"   # 多个一起，各出各的文件夹 + index.mdyueying "https://www.bilibili.com/video/BVxxxx" --all                      # B站 分 P / 合集 全部处理yueying --install-skill                                                    # 装 Claude Code 技能

默认输出 ./yueying_out/<视频名>/，日志和 report.md 默认中文（--ui-lang en 切英文）。常用参数：--lang zh 指定语言、--model small 换小模型（auto 自动选）、--device cpu、--interval 3 抽帧间隔、--frames 30 最多帧数、--scene 0.2 场景灵敏度、--no-dedupe 不去重、--no-asr 只要画面、--no-frames 只要文字、--force-asr 有字幕也识别、--cookies-from-browser chrome、--keep 保留原视频、--json 末尾打印一行 manifest。完整说明见上方英文表格。