vid2context

vid2context

[npm] [License: Apache-2.0] [Node 22+] [Ask DeepWiki]

Turn any video into context for AI agents. An MCP server, CLI, HTTP API and library that gives Claude, ChatGPT, Cursor, VS Code Copilot and any other agent the transcript, metadata and chapters of a video, playlist, podcast or media file, with timestamps it can cite.

Paste a video link or drop in a media file. vid2context returns the transcript, metadata and chapters in a form your agent can read and cite: plain text with [m:ss] timestamps, JSON, Markdown, SRT, VTT, or one Markdown context pack for many videos that you can upload to ChatGPT Projects, Claude Projects or a custom GPT.

It is free first. Local models run first, then hosted free tiers with your own key, then keyless hosts that need no account. Paid APIs are used last, and only when you set their key. With zero configuration it already works.

$ vid2context https://www.youtube.com/watch?v=jNQXAC9IVRw --timestamps[0:01] All right, so here we are, in front of the elephants[0:05] the cool thing about these guys is that they have really...[0:07] really really long trunks[0:12] and that's cool

Why

Research happens on video. A playlist explaining which used car to buy, a conference talk, a course, a podcast. Agents cannot watch any of it. vid2context turns those videos into text with timestamps, so an agent can absorb a whole series and answer "which model year should I buy?" while citing the episode and the moment that said so.

Design rule: raw transcripts are always kept. Summaries, briefs and extractions are an added layer, never a replacement.

How it works

For every input, vid2context resolves the source, fetches metadata and chapters, then walks a fallback chain until it has a transcript:

1. Creator-uploaded captions
2. Platform auto-captions
3. Download the audio and run speech-to-text
4. Video model mode, where a multimodal model watches the video so on-screen text, slides and code are captured too (planned)

Each speech-to-text and LLM step runs through a rung chain. A rung is one provider, one model and one credential. Rungs are tried in tier order and the chain falls through on errors, rate limits and outages, with cooldowns so a failing rung is skipped for a while. Every attempt is recorded, so you can always see what was tried and why it succeeded or failed:

Attempts:  resolve  ok      youtube  youtube:jNQXAC9IVRw  metadata ok      youtube  captions ok      youtube-captions  creator-captions

Status

First release (0.1.0), on npm as vid2context-mcp and vid2context. Everything below is built and covered by tests that replay real recorded responses.

Planned: video mode, where a multimodal model watches the whole video. The --mode video flag exists, but no provider implements it yet.

Quick start (zero keys)

Requires Node.js 22 or newer.

npx vid2context "https://www.youtube.com/watch?v=jNQXAC9IVRw" --timestamps

From source:

git clone https://github.com/nirholas/vid2context.gitcd vid2contextnpm installnpm run cli -- "https://www.youtube.com/watch?v=jNQXAC9IVRw" --timestamps

A local audio or video file works too. With no keys set, it is transcribed by the keyless OVH Whisper rung:

npm run cli -- ./interview.mp3 --format json > interview.json

More in docs/examples: runnable examples for the library, CLI, HTTP API, MCP, LangChain and Python, each a file in examples/ whose output on the page is checked by the tests.

CLI

vid2context <input> [flags]            same as `vid2context get <input>`vid2context <command> [args] [flags]
  get         Transcript and metadata for one video (the default command)  list        Expand a playlist, channel, feed or page into video refs  pack        One context pack (Markdown for an AI model) from one or many videos  bundle      A .zip with every format, the context pack and a README for agents  search      Keyword search over transcripts saved in the library  library     Saved video contexts  collection  Named groups of videos in the library  models      Every rung by tier, whether it is enabled, and the env vars it needs  doctor      Check Node, ffmpeg, yt-dlp, GPU, local servers, data directory, keys and network  config      Resolved settings and where each came from, secrets masked  serve       Start the HTTP API  mcp         Start the stdio MCP server  formats, completion, help

Common flags: -f, --format text|json|markdown|srt|vtt|pack, -t, --timestamps, --mode auto|captions|stt|video, --lang <code>, --policy free|local-only|quality, --only and --skip (rung ids or services), --json, -o, --out <file>, --verbose. Run vid2context help <command> for the rest. Full reference: docs/cli.md.

Program output goes to stdout and everything else to stderr, so vid2context URL > transcript.txt stays clean. On failure the CLI prints the error code, a hint on how to fix it, and every attempt that was made, and exits with a code per failure kind (3 unsupported input, 4 not found, 6 no transcript, 9 rate limited, 10 auth, and so on).

Use it from your agent

The MCP server is the npm package vid2context-mcp, listed in the MCP Registry as io.github.nirholas/vid2context. It needs no keys.

[Install in VS Code] [Install in VS Code Insiders] [Install in Cursor]

Claude Code

claude mcp add --scope user --transport stdio vid2context -- npx -y vid2context-mcp

Codex, Gemini CLI

codex mcp add vid2context -- npx -y vid2context-mcpgemini extensions install https://github.com/nirholas/vid2context

Claude Desktop, Cursor, Windsurf, Cline and other clients

{  "mcpServers": {    "vid2context": { "command": "npx", "args": ["-y", "vid2context-mcp"] }  }}

To pass a key, add "env": { "GROQ_API_KEY": "..." } to the entry. Agents setting this up for you can follow llms-install.md.

From source

From the repo root:

claude mcp add vid2context -- "$PWD/node_modules/.bin/tsx" "$PWD/packages/mcp/src/bin.ts"

Tools

Ask your agent something like "get the transcript of https://youtu.be/jNQXAC9IVRw and summarize it with timestamps". The server exposes 28 tools:

It also ships the prompts brief, buyer_guide and digest, and resources for saved videos, transcripts and collections (collections are subscribable). Failures come back as tool errors carrying the error code, hint and attempts, so the agent can explain the problem or retry with another mode. Details: docs/guides/mcp.md.

ChatGPT and Claude.ai

These connect only to remote MCP servers over HTTPS. Two options:

• Context pack: vid2context pack <url>... writes one Markdown file for one or many videos, sized for upload to a ChatGPT Project, Claude Project or custom GPT. Zero setup.
• Remote MCP: run the Streamable HTTP server with a bearer token or the built-in single-owner OAuth 2.1 server, behind a Cloudflare Tunnel so requests to YouTube still come from your own connection. See docs/guides/remote-mcp.md and docs/guides/cloudflare-tunnel.md.

Add free keys (optional)

Everything is optional. Each key you add unlocks more rungs, and the chain uses them before falling back to keyless hosts. Set them in your shell or a .env file.

Speech-to-text

LLM

Every model id can be overridden with <PROVIDER>_MODEL (a comma-separated list, in order) for LLMs, or <PROVIDER>_STT_MODEL for speech-to-text. To pool several free keys for one provider, put the extras in <PROVIDER>_FALLBACK_KEYS, for example GROQ_FALLBACK_KEYS=key2,key3. Each key becomes its own rung.

Chain controls

Architecture

TypeScript, ESM and npm workspaces, so the same code runs on Node, Cloudflare Workers, Vercel and in the browser.

packages/  core/        Contracts, rung chain, pipeline, formatters, chunking, caption cleanup, long audio.               Runtime-neutral, no Node-only imports.  registry/    Per-runtime provider registries (node, workers, browser), generated on install.  cli/         The vid2context command.  mcp/         The stdio MCP server; mcp-http/ serves it remotely, mcp-apps/ and mcp-agentic/ add               the viewer, sampling and elicitation.  server/      HTTP API on Hono with OpenAPI, for Node, Workers, Vercel and containers.  web/ chat/ extension/   Web app, chat app and browser extension.  ask/ insights/ translate/ chapters/ search/ library/ ...   Intelligence and library features.  discovery/   npm bundles, MCP Registry and directory manifests (server.json, glama.json, ...).providers/     One package per source, speech-to-text, LLM, embedding or storage backend               (youtube, ytdlp, bilibili, media-file, stt-whisper-cpp, llm-local, storage-sqlite, ...).scripts/  gen-registry.mjs         Scans providers/*/package.json and writes the generated registries.

Key decisions:

• Providers are separate packages. Each declares the runtimes it supports in its package.json ("vid2context": {"runtimes": [...]}) and default-exports a provider. npm run gen builds one registry per runtime, so Node-only providers never end up in a Workers or browser bundle.
• Plain fetch over vendor SDKs. Most speech-to-text and LLM APIs are OpenAI-compatible HTTP, which keeps providers edge-compatible and dependencies few.
• Providers never read process.env or the global fetch. Everything comes through the ctx object passed to each call, so the same provider works in every runtime and in tests.
• Formats live in one map. Every surface lists formats from formatters in packages/core/src/format, so a new formatter appears in the CLI, MCP and API at once.
• stdout is for output only. Logs go to stderr, which keeps CLI pipes and the MCP stdio protocol clean.

Start reading at packages/core/src/types.ts, packages/core/src/provider.ts and packages/core/src/chain.ts.

Development

npm install          # also generates the provider registriesnpm run typecheck    # tscnpm run lint         # Biomenpm test             # vitest, against real recorded fixturesnpm run test:live    # LIVE=1: hits the real servicesnpm run check        # typecheck + lint + testnpm run cli -- <url> # run the CLI from sourcenpm run mcp          # run the MCP server on stdionpm run examples:docs # rewrite docs/examples from the scripts in examples/

Tests use real responses recorded from the live services, never invented data. Live tests run only with LIVE=1.

Adding a provider

1. Create providers/<name>/ with a package.json that names the package @vid2context/provider-<name>, depends on @vid2context/core, and declares "vid2context": {"runtimes": [...]}.
2. Default-export a provider built with defineProvider from src/index.ts. Read config and make requests only through ctx.
3. Add tests under providers/<name>/test/ with a recorded fixture, plus a live test gated by LIVE=1.
4. Run npm install (or npm run gen) to pick it up in the registry. Never hand-edit the generated files.

Roadmap

Next up: video mode (a multimodal model watches the video, so on-screen action is captured as well as speech), Gemini and Deepgram or AssemblyAI speech-to-text, and podcast feeds with podcast:transcript. DRM services are out of scope. The full plan and what each work item covers is in docs/ROADMAP.md.

FAQ

How do I give Claude, ChatGPT or Cursor a YouTube video? Add the MCP server (above) and paste the link into your chat. The agent calls get_transcript and gets the transcript with [m:ss] timestamps. ChatGPT and Claude.ai need the remote MCP server or a context pack upload (see ChatGPT and Claude.ai).

Does it need an API key? No. Creator captions and auto-captions need no key, and speech-to-text falls back to keyless hosts. Keys only add faster or local options.

What if a video has no captions? vid2context downloads the audio and runs speech-to-text, preferring a local Whisper server, then free tiers such as Groq, then keyless hosts.

Can I keep AI processing on my machine? Point it at a local speech-to-text server and a local LLM and set VID2CONTEXT_POLICY=local-only. No hosted AI service is used; fetching a YouTube video still needs the network, a local file does not.

Is it a YouTube transcript API? Yes: a library, a CLI and an HTTP API with OpenAPI (vid2context serve, see docs/guides/http-api.md). It returns text, JSON and timestamps, keeps the raw transcript, and records every provider it tried.

Does it work on long videos and whole playlists? Yes. Agents read the outline first and fetch only the chapters they need (about 12k tokens per hour of speech), long audio is split and transcribed in parallel, and list_items or vid2context list expands a playlist, channel or feed into its videos.

License

Apache-2.0