AI Video Generation

You are executing generation for an approved ai_video storyboard. The workflow file workflows/ai_video.json lists this as the generate step, following ai-video-plan. The storyboard — scenes[], imageRefs[], styleAnchor — was written by the planning skill; your job is to generate clips and audio from it.

This skill covers Phase 6 (scene generation dispatch) and Phase 7 (draft-phase regenQueue processing). For Phases 0-2 (story clarification, storyboard writes, review), see skills/ai-video-plan/SKILL.md.

Sub-skills

Name	Path	When to load
eval-scenes	skills/eval-scenes/SKILL.md	After generating scenes in Phase 6 — quality evaluation + retry loop

---

Status machine

pending → storyboard_ready → draft → final

Transition	Writer	Precondition
pending → storyboard_ready	ai-video-plan (agent)	Phase 1 writes complete: imageRefs anchors+images filled, styleAnchor written, scenes[] populated. tracks[0].items still [].
storyboard_ready → storyboard_ready	ai-video-plan (agent) in review-phase	User asks for storyboard changes via chat. Mutate project.json; status stays storyboard_ready.
(implicit, UI-driven)	UI	User clicks Approve. UI writes storyboard.approval = {approvedAt: <ISO8601>}. Status stays storyboard_ready — you observe the field and start Phase 6.
storyboard_ready → draft	you (this skill)	Every storyboard.scenes[i] has a matching clip in tracks[0].items (either generation.sceneId === scene.id OR a batchShots[] entry for it).
draft → final	human (via UI)	Manual review. Out of scope for this skill.

---

Phase 6 — Scene generation (status: storyboard_ready, storyboard.approval is set)

Entry point — how you get here

You are not file-watching. You enter Phase 6 because the user tells you to in chat. The typical message looks like:

"I approved the storyboard for project "X". Please proceed with scene generation per the ai-video-generate skill Phase 6 contract."

That message is produced by either:

• The UI's post-approval "Tell your agent" panel (a one-click "Copy" button on ApproveAndGenerate), or
• The montaj approve CLI command (which prints the same message for the user to paste), or
• The user typing it manually (e.g. they edited project.json directly and now want you to go).

Verification guard — always run before generating

Before calling kling_generate, re-read project.json and verify:

1. project.projectType === "ai_video".
2. project.status === "storyboard_ready".
3. project.storyboard.approval exists AND has an approvedAt field.
4. project.storyboard.scenes is non-empty.

If ANY of those is false, do not proceed. Tell the user what's missing and ask them to click the Approve button in the UI (or run montaj approve). Common failure modes:

• User said "I approved, go" but storyboard.approval is absent → the UI save failed, or the user saw a stale view. Ask them to click Approve again.
• Status is pending → storyboard hasn't been written yet. Return to Phase 1.
• Status is already draft → nothing to do. Tell them.
• scenes is empty → Phase 1 never populated the plan. Return to Phase 1 instead of generating from thin air.

Never trust the user's chat message alone as the authorization signal. The file field is the source of truth — the message is just the trigger to check.

Once the guard passes, start generating.

Step A — determine the scene set to generate

Iterate storyboard.scenes[]. For each scene, check for a matching clip on tracks[0].items:

• Single-shot origin: tracks[0].items.some(c => c.generation?.sceneId === scene.id).
• Batched origin: tracks[0].items.some(c => c.generation?.batchShots?.some(shot => shot.sceneId === scene.id)).

A scene is "already done" if EITHER check passes. Then:

• Clip exists → skip (user is re-approving after editing only a subset).
• scene.lastError set AND no clip → retry.
• Else → new generation.

This makes generation idempotent and incremental:

• Full approval (first run): all scenes generate.
• Re-approval after editing specific scenes: only the changed scenes regenerate. Unchanged scenes keep their existing clips.
• Selective regeneration (user asks to redo specific scenes): remove the clip for that scene from tracks[0].items, then call kling_generate --project-id X --scene-id Y. The step's dedup guard replaces any existing clip for the same sceneId, so you can also just re-run the step — it overwrites.

Never regenerate all scenes when only some changed. Each Kling call costs credits and takes ~60s. If the user edited scenes 1 and 3, generate only those two. The agent should diff the user's changes against the current storyboard to determine which scenes need regeneration.

Step B — pick a dispatch mode

Three modes; infer from the prompt + scene count (ask in Phase 0 if unclear):

Independent (default — parallel)

Each scene is a separate kling_generate call, self-contained. You MUST fire these in parallel — call all scenes' kling_generate tool calls in a single response so they execute concurrently. Cap at 4 concurrent calls; if there are more than 4 scenes, fire 4 at a time, wait for any to complete, then fire the next. Do NOT generate scenes one at a time in a sequential loop — that wastes minutes of wall-clock time that parallel dispatch avoids.

When parallel results land out of narrative order, compute each clip's start/end from the scene's position in storyboard.scenes[], not from generation order. Fault-isolated and easy to regenerate.

How to fire in parallel: Include multiple tool calls in one assistant message. For example, if you have 5 scenes and are capping at 4 concurrent:

1. First message: call kling_generate for scenes 1, 2, 3, 4 simultaneously (4 tool calls in one response).
2. When results arrive, write all 4 clips to tracks[0].items.
3. Second message: call kling_generate for scene 5.
4. Write the final clip and check if all scenes are done → set status to draft.

Chained continuity

Scene N visually continues N-1. Generate sequentially: after N-1 succeeds, call snapshot --input <clip.src> --at <clip.outPoint> --out <frame.jpg> to extract its last frame, then call kling_generate --first-frame <frame.jpg> ... for scene N. Slower (no parallelism), but preserves visual continuity at scene boundaries.

Batched (multi-shot)

Group up to 6 scenes into ONE kling_generate call using --multi-shot --shot-type customize --multi-prompt '<json>'. Kling returns a single concatenated video. Cheaper (1 billing unit per batch vs N), stronger visual consistency across shots (shared stylistic context), one poll instead of N.

Trade-offs:

• Per-shot prompt cap is 512 chars, not 2500. Write tighter per-scene prose.
• No --first-frame / --last-frame — frame control isn't supported. Incompatible with chained.
• All-or-nothing failure. If the batch fails on one scene, the whole batch is lost. Regenerating a single scene from a completed batch runs as a single-shot call spliced into tracks[0].items.
• One clip per batch, not per scene. Store per-scene mapping in generation.batchShots[] (see Step D).
• Storyboard > 6 scenes → split into multiple batches.

Picking a mode

• Default for nearly everything → independent (parallel). Fastest wall-clock time, fault-isolated, no prompt-length restrictions. Use this unless one of the below specifically applies.
• "Flowing" / "seamless" transitions requiring frame continuity (e.g. a continuous camera move across scenes) → chained. Only use when the creative brief specifically demands visual continuity at scene boundaries.
• Strong scene-to-scene continuity AND scenes fit in 512 chars each AND you want cheapest billing → batched. Best cost-to-quality when applicable.

State your chosen mode in chat once at the start — the user can redirect if wrong.

Picking a model

Two Kling models are available. Pass --model <name> to kling_generate.

• kling-video-o1 (preferred for visual quality) — newest model. Only 5s or 10s durations. No multi-shot. End frame (--last-frame) requires --mode pro. Does NOT generate audio — clips are silent.
• kling-v3-omni (required for audio) — flexible 3–15s durations, multi-shot support, start+end frame in both std/pro. Generates audio when sound: "on". Use when scenes have dialogue or need sound.

How to decide: The model is per-scene, not per-project — you can mix and match. The step auto-upgrades to o1 when safe:

• Duration is 5 or 10 AND sound: "off" → auto-upgrades to kling-video-o1.
• Duration is 5 or 10 AND sound: "on" → stays on kling-v3-omni (needs audio).
• Duration is anything else → kling-v3-omni.

This lets you get the best quality where possible while keeping flexible pacing elsewhere. The connector snaps invalid durations to the nearest allowed value, but snapping changes your editorial pacing — better to pick the right model per scene than rely on snapping.

State your model choices in chat alongside the dispatch mode. Record the actual model used on generation.model for each clip.

Transition style — hard cuts by default

Each scene is generated from scratch via text-to-video. Shot-to-shot transitions are hard cuts. Do NOT use --first-frame to chain scene N's last frame into scene N+1's generation — this produces a morphy, dissolve-y feel that reads as AI-generated.

The only exception is a deliberate match-cut — where the end of scene A and the start of scene B form an intentional visual rhyme (e.g. a ball rolling → a globe spinning). This is rare: 0-1 times per video, only when the user explicitly requests it. If you're unsure whether something qualifies as a match-cut, it doesn't — use a hard cut.

Identity consistency across hard cuts comes from the same character specs and ref images being passed to every scene — not from pixel handoff between frames. The kling_generate step appends character descriptions and <<<image_N>>> tokens to every prompt, so identity holds via the spec.

Note: Chained dispatch mode (documented above) remains available for the rare match-cut case. It is no longer the recommended default — use independent dispatch with hard cuts unless the user specifically requests visual continuity at a scene boundary.

Step C — prompt composition (handled by the step)

In project-aware mode (--project-id + --scene-id), the kling_generate step handles all prompt composition automatically. The agent does NOT compose the wire prompt manually. The step:

1. Reads the scene's ## Section prompt and flattens it (Subject → Action → Dialogue → Setting order, period-terminated).
2. Prepends storyboard.styleAnchor as a short prefix.
3. Places <<<image_N>>> tokens inline at character/object label mentions (matches on first word of label for flexibility — "Rosie" matches even if label is "Rosie the Dog").
4. Appends [SHOT SCALE] and [CAMERA MOVE] tags from the scene's structured fields.
5. Auto-sets a default negative prompt targeting common Kling failure modes.
6. Generates a random seed for reproducibility.
7. Auto-upgrades to kling-video-o1 when duration is 5/10 and sound is off.

The agent's only job is writing good ## Section prompts in Phase 1. Everything else is mechanical.

What gets stored on generation.prompt: the composed wire-ready string (with tokens, style anchor, camera tags). This is the exact prompt sent to Kling.

• Length caps (enforced by connector):

  • Single-shot: silently truncates at 2500 chars. Keep scene prompts under ~100 words to stay safe.
  • Multi-shot customize: hard-rejects any multi_prompt[i].prompt > 512 chars.

• Resolve scene.refImages IDs against storyboard.imageRefs (use imageRefs[i].refImages[0] as the primary path). Enforce the API cap of 7 refs per scene.

• Respect Kling's length cap: 2500 chars in single-shot, 512 chars per shot in multi-shot.

Step D — call and write

Single-shot (independent or chained)

kling_generate \
  --prompt <combined> \
  --duration <scene.duration> \
  --aspect-ratio <storyboard.aspectRatio> \
  --model <chosen model> \
  --ref-image <path> [--ref-image <path> ...] \
  --out <path> \
  --external-task-id <scene.id>

Add --first-frame <path> for chained mode (N-1's last frame).

On success: append a new clip to tracks[0].items:

{
  "id": "clip-<scene.id>",
  "type": "video",
  "src": "<returned path>",
  "start": <cumulative sum of prior durations>,
  "end": <start + scene.duration>,
  "inPoint": 0,
  "outPoint": <scene.duration>,
  "generation": {
    "sceneId": "<scene.id>",
    "provider": "kling",
    "model": "<chosen model>",
    "prompt": "<combined>",
    "refImages": ["<ref_id>", ...],
    "duration": <scene.duration>,
    "attempts": []
  }
}

These clips have no editing proxy, and neither does the one kling_generate writes for you. lib/ai_video.py:235-253 builds the item with id/type/src/start/end/inPoint/outPoint/generation/sourceDuration and no proxySrc, so an ai_video project fails the WebCodecs eligibility gate on every clip (montaj_assets/editor/src/engine/eligibility.ts:69 refuses the whole project when any track-0 video item lacks the field). Do not invent a value for it and do not compute the path yourself — Step E.2 below backfills them all in one call, after every scene has landed.

The prompt stored on generation.prompt is the caller's composed string (styleAnchor + scene prose in natural language) — NOT the wire string the connector produced after prepending its ref clause. The connector derives the ref clause deterministically from refImages, so regen can re-run the same caller prompt and reproduce the same wire string. Phase 7's regenerate flows (full-scene and subcut) pre-fill the prompt field from this.

Post-download normalization: After a clip is saved, save_clip_to_project automatically probes it and normalizes to the project's resolution and fps (from project.settings) if they don't match. For example, Kling outputs 1280x720 H.264 clips, which are upscaled to the project resolution (typically 1920x1080). The normalized file is written alongside the original (*_normalized.mp4) and clip.src is updated to point to it. If normalization fails, the original clip is kept as-is.

Write project.json back IMMEDIATELY after each scene completes — do not batch writes. The UI watches for changes via SSE and flips scene chips from "pending" → "done" in real time. If you wait until all scenes finish to write, the user sees no progress for minutes. Each kling_generate call returns → append the clip to tracks[0].items → save project.json → move to the next result. When running scenes in parallel, save after EACH result lands (not after all parallel calls complete).

On failure: record storyboard.scenes[i].lastError = {ts: <ISO8601>, message: <error>} and write project.json back immediately. Do NOT append to tracks[0].items. Continue to the next scene. The UI updates in real-time via SSE.

On retry after failures: Before retrying failed scenes, clear lastError on each scene you're about to retry — set storyboard.scenes[i].lastError = undefined and write project.json. This resets the UI's red "failed" chips back to "pending" so the user sees live progress. Then proceed with generation as normal. On success, the clip write naturally clears the failed state in the UI.

Batched (multi-shot customize)

Build multi_prompt JSON from the batch:

[
  {"index": 1, "prompt": "<combined for scene_A>", "duration": "3"},
  {"index": 2, "prompt": "<combined for scene_B>", "duration": "4"},
  ...
]

Each prompt is the combined styleAnchor + scene prose + inline <<<image_N>>> tokens (SAME composition as single-shot, but respect the 512-char per-shot cap). Call:

kling_generate \
  --multi-shot \
  --shot-type customize \
  --multi-prompt '<json>' \
  --aspect-ratio <storyboard.aspectRatio> \
  --ref-image <path> [--ref-image <path> ...] \
  --out <path> \
  --external-task-id batch-<first_id>-<last_id>

Refs passed apply to any shot in the batch. Cap still 7 total.

On success: append ONE clip to tracks[0].items representing the whole batch:

{
  "id": "batch-<first_scene_id>-<last_scene_id>",
  "type": "video",
  "src": "<returned path>",
  "start": <cumulative>,
  "end": <start + total_batch_duration>,
  "inPoint": 0,
  "outPoint": <total_batch_duration>,
  "generation": {
    "provider": "kling",
    "model": "<chosen model>",
    "multiShot": true,
    "shotType": "customize",
    "refImages": ["<ref_id>", ...],
    "attempts": [],
    "batchShots": [
      {"sceneId": "scene_A", "index": 1, "prompt": "<combined_A>", "start": 0.0, "end": 3.0, "duration": 3},
      {"sceneId": "scene_B", "index": 2, "prompt": "<combined_B>", "start": 3.0, "end": 7.0, "duration": 4}
    ]
  }
}

Same proxy caveat as the single-shot clip above: a batched clip carries no proxySrc either. Step E.2 covers both.

batchShots[i].start / end are relative to the batch clip, not the project timeline. The UI uses these for per-scene progress.

On failure (batch-level): the whole batch is lost. Record storyboard.scenes[i].lastError = {ts, message, batchId} on EVERY scene in the batch. Do NOT append to tracks[0].items. The user can re-click Approve (Step A skips scenes with existing clips — retry is automatic for batches with no clip) or edit individual prompts and retry.

Step E — Audio generation, then wrap up

Important: generate audio BEFORE setting status to draft. Setting draft routes the user to ReviewView — audio tracks must already be on the project by then.

After all scenes have clips on tracks[0].items, process the audio intake fields first (Step E.1), then set status (Step E.2).

Step E.1 — Audio generation and assembly

Process the audio intake fields. Compute total video duration:

total_duration = sum(scene['duration'] for scene in storyboard['scenes'])

Re-run cleanup. Phase 6 may run multiple times. Before appending any generated tracks, remove prior Phase-6-generated audio:

project['audio']['tracks'] = [
    t for t in project['audio'].get('tracks', [])
    if not (t['id'] == 'voiceover' or t['id'].startswith('music-') or t['id'] == 'music')
]

Music

If storyboard.music is set:

Upload mode (storyboard.music.mode === 'upload'):

• Probe the file: run_step('probe', { 'input': storyboard.music.path }) → get duration.
• Append one AudioTrack:

  {
    "id": "music",
    "src": "<storyboard.music.path>",
    "start": 0,
    "end": min(duration, total_duration),
    "sourceDuration": duration,
    "volume": 0.3,
    "label": "music (uploaded)",
    "ducking": { "enabled": true }
  }
  

Describe mode (storyboard.music.mode === 'describe'):

• Call run_step('generate_music', { prompt: storyboard.music.prompt, out: '<project_dir>/assets/music.wav' }).
• Lyria Clip produces ~30s. If total_duration > duration, tile the track by creating multiple AudioTrack entries pointing to the same file at sequential start offsets:

  start = 0
  while start < total_duration:
      seg_end = min(start + dur, total_duration)
      append AudioTrack with id=f"music-{start}", src=result.path,
        start=start, end=seg_end, inPoint=0, outPoint=seg_end-start,
        sourceDuration=dur, volume=0.3, label=f"music (generated, loop at {start}s)",
        ducking={ enabled: true }
      start += dur
  

Ducking config: { enabled: true } is sufficient. The render pipeline's mix-audio.js applies defaults for depth (−12 dB), attack (0.3s), and release (0.5s) when those fields are absent.

Voiceover

If storyboard.voiceover is set:

Step 1 — decide script vs. brief. Inspect storyboard.voiceover.prompt:

• If the text reads like literal spoken lines (first-person narrative, quoted dialogue) → use verbatim as TTS input.
• If the text reads like a direction ("narrate like a documentary") → expand into a full script via LLM call, sized to ~total_duration * 150 / 60 words (~150 wpm narration pace).
• Ambiguous cases: prefer verbatim (trust the user's text).

Concrete examples:

User prompt	Interpretation	Action
"Welcome to our farm, where every morning begins with..."	First-person narrative, reads as spoken lines	verbatim — feed directly to TTS
"narrate like David Attenborough describing a quiet morning"	Clear direction, no content	expand — LLM generates a script in that voice
"the dog looks up at the sky"	Third-person description, could go either way	verbatim (prefer trusting the user)
"make it sound urgent and dramatic, mention the storm"	Direction + content hint	expand — LLM writes an urgent script about a storm

Step 2 — call the step (with Kling→Gemini fallback):

Voice selection. TTS_VOICES in connectors/kling.py is currently empty (placeholder IDs). Pass the raw voice string directly — the connector falls back to using it as-is via TTS_VOICES.get(voice, voice). For Gemini, use documented voice names.

Inferred tone	Kling --voice	Gemini --voice
Neutral / documentary / instructional	"female_warm" (raw string; Kling resolves or uses default)	"Kore"
Energetic / commercial / dramatic	"female_warm"	"Puck"
Dark / serious	"male_calm"	"Charon"

Always record the chosen voice on the resulting track's label (e.g. "voiceover (kling:female_warm)" or "voiceover (gemini:Kore)").

out = f"{project_dir}/assets/voiceover.wav"

# Primary: Kling TTS. Fall back to Gemini if Kling errors.
try:
    result = run_step('generate_voiceover', {
        'text':   script,
        'voice':  'female_warm',       # raw string — Kling connector passes through
        'out':    out,
        'vendor': 'kling',
    })
    voice_label = 'kling:female_warm'
except Exception as e:
    # Kling TTS failed (likely due to placeholder voice IDs) — retry with Gemini.
    agent_log(f"Kling TTS failed ({e}); retrying with Gemini TTS")
    result = run_step('generate_voiceover', {
        'text':   script,
        'voice':  'Kore',              # Gemini documented voice name
        'out':    out,
        'vendor': 'gemini',
    })
    voice_label = 'gemini:Kore'

Step 3 — append as track:

{
  "id": "voiceover",
  "src": "<result.path>",
  "start": 0,
  "end": min(result.duration_seconds, total_duration),
  "sourceDuration": result.duration_seconds,
  "volume": 1.0,
  "label": "voiceover (<voice_label>)",
  "ducking": { "enabled": false }
}

Duration mismatch handling:

• VO duration > total_duration: clamp end to total_duration (truncates tail). Warn the user.
• VO duration < total_duration: VO plays and stops; silence fills the remainder (with music still playing if present). No warning needed.

Error handling: If generate_music or generate_voiceover fails, skip the failed track — do not abort the whole project. Surface the error to the user. Continue with the other track if available.

Write project.json after appending the audio tracks.

Step E.2 — Set status

• When every storyboard.scenes[i] has a matching clip (by sceneId OR batchShots sceneId) AND audio generation is complete (or skipped if no intake fields): first backfill the editing proxies, then set project.status = "draft". The UI's EditorPage routing carries the user to ReviewView.