4k Vfx

4k-vfx

Maps a video-to-video VFX move onto the Pika MCP's Seedance 4K reference-to-video. The user hands you a clip + the change they want; you read EVERY frame (locally extracted with ffmpeg and tiled into contact sheets) and understand the audio (speech + music + SFX + ambience), author a Seedance-faithful prompt that locks the original (face, gestures, camera move) and time-codes the change to the right beat, then re-render the same shot in 4K with the VFX baked in. One run produces one 4K clip for one requested change.

How it works — why reading EVERY frame (and the audio) is the mechanism (do not skip the reading pass)

Seedance treats the reference video as a motion/style anchor, NOT a pixel-locked base — it does not copy the input frame-for-frame, it re-generates the shot guided by your prompt. So fidelity comes from the PROMPT: the more exhaustively the prompt describes the original (subject identity, exact gestures, camera move, framing, lighting, palette, wardrobe) and the audio (dialogue + lip-sync, music/SFX beats), the more the 4K output reads as the same shot with the change layered on. Reading EVERY frame — not a sample — is how you build that exhaustive description: you locally extract all frames with ffmpeg, tile them 25-to-a-contact-sheet (5×5), and analyze every sheet so the prompt is built from the entire timeline rather than a handful of stills. Skip the read (or read only a sample) and the output drifts — a different face, a different camera move, a different room, a missed motion beat. The reading pass — all frames + the audio — IS the skill.

Prerequisites

pika MCP available, and local ffmpeg in the run environment (used to extract every frame and build the contact sheets). Two inputs from the user: (1) a video clip (a local file or a public URL) and (2) the VFX change they want made to it. Tools used: upload_asset, probe_media, analyze_media, transcribe_audio, estimate_cost, generate_reference_video, task_status (frame extraction + tiling is done locally with ffmpeg, not an MCP tool).

Stage 0 — Gather inputs (settle this first)

You need exactly two things before generating:

1. The video — the original clip to transform. If they gave a local file, you will upload it (Step 1); if they gave a public URL, keep it.
2. The change / VFX — what should be different in the output (e.g. "turn the plaza into an open desert on a finger snap", "make it snow", "set the room on fire behind me", "morph my jacket into glowing armor").

If either is missing, ask for it before doing anything else — do not invent a change, and do not proceed on a clip you cannot reach. Once both are in hand, say in one line what you're about to do ("Reading every frame of your clip to build the desert-on-snap 4K prompt…") and run Steps 1–5 without further check-ins — but stop at the Step 6 agreement gate before generating: the 4K render is the expensive, irreversible step, and it only fires after the user approves the prompt and cost.

Step 1 — Upload the clip (upload_asset)

Real video must be uploaded — inline base64 only works for tiny assets (<~3MB), so any actual clip goes through the presigned-upload path.

1. Call upload_asset with filename, mime_type (video/mp4 for mp4), and size_bytes (the file's byte size).
2. PUT the raw bytes to the returned presigned_url.
3. Keep the returned public_url → state.clip_url.

If the user already gave a public URL, skip this step and set state.clip_url to it.

Step 2 — Probe (probe_media)

Call probe_media on state.clip_url to read duration, fps, dimensions, and aspect ratio. Use these to: (a) sanity-check the all-frame extraction in Step 3 (duration × fps ≈ total frame count ≈ 25 × number of sheets), and (b) choose the output aspect_ratio and duration here, matched to the source — the Step-6 gate quotes this choice to the user and Step 7 fires with it.

Step 3 — Extract EVERY frame locally and tile into contact sheets (ffmpeg)

Read the whole video, not a sample. Use local ffmpeg to extract every frame at the source rate (no drops, no exceptions), then tile them into legible contact sheets at 25 frames per sheet (5×5 grid) so each analyze_media call covers 25 consecutive frames at once.

Work in a relative working directory (do not use an absolute system temp path):

mkdir -p frames sheetsffmpeg -i INPUT -vsync 0 frames/f_%05d.png            # every frame, no drops

Then build the 25-per-sheet contact sheets (5×5), scaling each frame down so the sheet is legible and stamping the frame index onto each tile:

ffmpeg -i INPUT -vf "scale=480:-1,drawtext=text='%{n}':x=4:y=4:fontsize=20:fontcolor=yellow:box=1:[email protected],tile=5x5" sheets/sheet_%03d.png

drawtext=text='%{n}' stamps the frame index on each tile so the analysis can reference exact frames; tile=5x5 groups 25 consecutive frames per sheet. If drawtext is unavailable in the local ffmpeg build, fall back to plain scale=480:-1,tile=5x5. INPUT is the local clip (or a local copy of the URL). Confirm the sheet count lines up with the probe in Step 2 — every frame must land on a sheet, no exceptions.

Step 4 — Read EVERY sheet + understand the audio (upload_asset, analyze_media, transcribe_audio)

This is the fidelity step, and it has two mandatory halves: read all the frames and understand the audio. Neither is optional.

4a — Read all the frames (every sheet)

For each contact sheet from Step 3 (cover ALL of them — every frame is on a sheet, no exceptions): upload it with upload_asset (the sheets are images, so upload to get a URL), then call analyze_media on that sheet URL with a query that reads the full progression across those 25 frames — pulling out everything the prompt must lock:

• Subject / identity — who/what is in frame; face, hair, build, distinguishing features.
• Exact motion & gestures — what the subject does, beat by beat, referencing the stamped frame indices (e.g. "raises right hand around frame 0048, snaps fingers around frame 0072").
• Camera move — static / pan / push-in / handheld / orbit, and its timing.
• Framing — shot size (close/medium/wide), subject position, headroom.
• Lighting — direction, hardness, color temperature, time of day.
• Palette — dominant colors, mood.
• Wardrobe / props / setting — clothing, objects, background.

Carry the read across all sheets so the entire timeline is understood start-to-end, not just a sampled middle.

4b — Understand the audio (mandatory, not just speech)

Always read the audio — it is required, not optional. Two passes:

• transcribe_audio for speech + timestamps (the words and when they're said), so the prompt can preserve dialogue and lip-sync beats.
• analyze_media on the audio (or the video URL) to understand the NON-speech audio too — music, sound effects, ambience, tone, and rhythm/beats.

Fold the audio understanding into the prompt where it matters — preserve dialogue + lip-sync if anyone speaks, and time the VFX change to an audio beat / sound cue when the change should land on the music or a sound (e.g. snap, hit, downbeat) rather than a bare timestamp.

Assemble the all-frame read and the audio read into a single faithful description of the original — that description is the raw material for Step 5.

Step 5 — Write the Seedance prompt (the authoring step, NEVER skip it)

Compose ONE prompt = a faithful, exhaustive description of the original (the locked elements) + the user's requested change, time-coded + a reference to the input clip via the @Video1 token. Be thorough: every detail you read in Step 4 that you omit is a detail Seedance is free to change.

Template — fill the [SLOTS] from your Step-4 read:

Re-render this exact shot @Video1 in cinematic 4K. LOCK the original: [SUBJECT/IDENTITY — face, hair, build, wardrobe], performing [EXACT GESTURES, beat by beat]. Keep the SAME camera move ([CAMERA: static / pan / push-in / handheld], [timing]), the SAME framing ([SHOT SIZE + subject position]), the SAME lighting ([direction, hardness, color temp, time of day]) and the SAME palette ([dominant colors / mood]). CHANGE: [the VFX], time-coded — at [t0]–[t1]s [what the scene looks like before the change], then at [t2]s [the change triggers / VFX appears] and [how the scene reads after]. Everything not described by the change stays identical to @Video1. Photorealistic, high detail, consistent identity throughout.

Time-code the change so Seedance knows when it happens relative to the locked motion (e.g. "at 0–3s the plaza is unchanged as the subject raises their hand; at 3s on the finger snap the plaza dissolves into an open desert, sand and heat-haze replacing the buildings while the subject, camera move, and framing stay identical"). The stronger and more specific the time-coded change clause, the less Seedance ignores it.

Step 6 — Agreement gate (estimate_cost) — MANDATORY before generating

The 4K render is the expensive step, and once fired it can't be un-spent. Never call generate_reference_video without explicit user approval in this session. Present, in one message:

1. The full Step-5 prompt — verbatim, so the user can catch a wrong detail (face, gesture, timing) before it costs money.
2. The generation recipe — provider seedance, model standard, resolution 4k, plus the aspect_ratio and duration you chose from the Step-2 probe.
3. The estimated cost — call estimate_cost for the Step-7 generate_reference_video call and quote the result. If the estimate errors, say the cost is unknown and flag that 4K standard is the top-price tier — do not silently skip the number.

Then wait. Three outcomes:

• Approve → proceed to Step 7 unchanged.
• Revise — the user corrects a detail or the change clause → update the Step-5 prompt (re-reading specific sheets if the correction demands it) and re-present the gate.
• Abort / no answer → do not generate. Never treat silence as approval.

Only skip this gate if the user explicitly pre-authorized the spend in this session ("just run it, don't ask", a standing instruction to render without confirmation). Having provided the clip and the change is NOT pre-authorization — that's just Stage 0 input.

Step 7 — Generate (Seedance 4K reference-to-video) (generate_reference_video)

Call generate_reference_video with the exact 4K recipe:

• provider: seedance
• seedance_model: standard — REQUIRED for 4k (fast/mini cap at 720p).
• resolution: 4k
• reference_videos: [state.clip_url] — the original footage as the motion/style anchor (Seedance accepts up to 3 videos).
• reference_images: optional — style/creature/identity anchors (e.g. a generated reference image), only if you have one.
• prompt: the Step-5 prompt (uses the @Video1 token to reference the input clip; no max length, so be exhaustive).
• aspect_ratio: auto (or match the source from Step 2).
• duration: matched to the source, OR set auto_duration: true.

Do NOT pass negative_prompt (not supported on seedance — the call is rejected) and do NOT pass any fps param (it doesn't exist).

Step 8 — Deliver

If the call returns inline, you have the 4K video URL. If it returns { task_id, status: running/background }, poll task_status(task_id) in a tight loop until completed / failed / cancelled. On completion, return the 4K video URL to the user. Note: the output CDN may be egress-blocked for in-session download — so deliver the URL itself rather than trying to fetch the bytes locally.

Failure modes

One-liner recap

Extract EVERY frame locally with ffmpeg → tile into 25-per-sheet (5×5) contact sheets and analyze every sheet → understand the audio (transcribe speech + analyze music/SFX/ambience) → write an exhaustive Seedance-faithful prompt that locks the original and time-codes the change to the right audio beat → agreement gate: show the prompt + recipe + estimate_cost and wait for approval → re-render the same shot in 4K via generate_reference_video (seedance / standard / 4k, original as reference_videos, @Video1 in the prompt, no negative_prompt).