video-extract-mcp

Give an AI agent any video link: download the file, read the transcript, or get just the frames that matter — all on your own machine.

Two jobs, and you can use either on its own:

Get the video. A YouTube link, a TikTok, a WeChat Channels share URL, a raw .mp4, or a page from a site nobody has heard of — it resolves and downloads it, whole or just the section you asked for. If that is all you need, stop there; nothing forces you to analyse anything.

Or read it. A transcript (real captions when the platform has them, local speech recognition when it does not) and a small set of important keyframes — deduplicated, scene-aware, and scored — instead of a thousand near-identical stills.

Built for AI agents. Three MCP tools, no cloud, no API keys, no Python.

---

What it can do

Get the video, from almost anywhere. Paste a link and it resolves: YouTube, TikTok, Instagram, X, Facebook and Reels, Twitch, Vimeo, Reddit, WeChat Channels, a bare .mp4 or .m3u8, or a page on a site nobody has heard of. Local files work too. Anything unsupported comes back as a clear failure rather than a crash.

From there you choose how much work to pay for:

• Just look it up. Title, creator, duration, chapter list, description, comment count — without downloading a byte. On a long video this is how you find the one section worth analysing.
• Download the file. The whole thing, or just seconds 300–420 of it. Where the platform supports ranged fetching, only that section is transferred rather than the full video.
• Get a transcript. Real captions when the platform has them, in any language it publishes; local speech recognition when it has none. The result says which you got. A captions-only request skips the download entirely — seconds instead of minutes.
• Get the frames that matter. Not every Nth frame: scene changes, on-screen text appearing, genuinely new visuals — deduplicated and scored, typically a few dozen for an hour of video. Or uniform sampling, or one exact frame at one exact second, if that is what you need.
• Do several at once. Pass a list of videos and get a result per video; one failing does not sink the rest.
• Reach private or rate-limited content. Point it at your browser's cookies and it can fetch age-restricted, members-only or followers-only media, and shrug off the bot checks that block anonymous downloads.
• Run long jobs in the background. A full analysis of an hour-long video takes minutes; it can run as a background task, report progress over a local HTTP endpoint, and hand back the result even if your client gave up waiting.

Everything happens on your machine. No API key, no upload, no third-party service.

Why this exists

Two problems, really. Getting the video at all — every platform hides its media behind a different mechanism, and none of them want a script fetching it. And then reading it: an LLM cannot watch a video, and the usual workaround — dump every Nth frame into the context window — burns enormous amounts of context on frames that are 98% identical to the one before, while still missing the slide that changed when nothing else moved.

video-extract-mcp handles both:

• Fetching, from almost anywhere. One resolver chain covers the big platforms, direct media URLs, and generic sites, with ranged fetches where the platform allows them and cookie support for anything that needs a login. Downloading is a complete use of this tool, not a step on the way to something else.

• Transcript, honestly sourced. The platform's own captions are used whenever the video has any — human-written first, otherwise the platform's automatic ones. Audio is transcribed locally (Whisper or SenseVoice) only for videos with no captions at all. The result tells you which you got, via transcript.source.

• Keyframes chosen, not sampled. Scene-boundary detection, blur/quality filtering, on-screen-text novelty (subtitle-aware, so burned-in captions don't preserve redundant frames), and image-embedding similarity feed an iterative diversity-aware selector.

• Output goes to disk, not into your context. The tool reply is a compact summary plus file paths. A 35-frame manifest and a full transcript don't belong in a conversation where the agent needs three numbers from them.

• Everything runs on your machine. No third-party API, no upload, no key. Long analyses can run as MCP background tasks — the tool returns a handle immediately and pushes progress; see Background tasks below.

Quick start

Install the system binaries first — these can't come from npm:

# macOS; use your package manager elsewhere
brew install ffmpeg yt-dlp tesseract tesseract-lang

Then point your MCP client at the package. There are two ways, and they differ in ways worth thirty seconds of your time.

Option A — npx, nothing installed. Simplest, and it picks up new releases on its own.

Claude Code:

claude mcp add --scope user video-extract -- npx -y @yanlinglabs/video-extract-mcp@latest

Codex:

codex mcp add video-extract -- npx -y @yanlinglabs/video-extract-mcp@latest

Another agent? Point it at SKILL.md and it can install itself.

Keep the @latest — without it npx pins to the first version it cached and never updates.

Option B — installed globally. Starts faster and gives you the video-extract status CLI as a real command.

npm install -g @yanlinglabs/video-extract-mcp

Then register it — Claude Code:

claude mcp add --scope user video-extract -- video-extract-mcp

Codex:

codex mcp add video-extract -- video-extract-mcp

	npx (A)	global install (B)
Updates	automatic only with @latest in the spec — a bare npx -y @yanlinglabs/video-extract-mcp pins to the first version it cached and never updates	manual: npm update -g @yanlinglabs/video-extract-mcp. You stay on the installed version until you run it
Startup	~0.9s (npm resolution on every launch)	~0.1s
video-extract status in your shell	not on PATH — needs npx -y -p @yanlinglabs/video-extract-mcp video-extract status	works directly
Working directory	must not be this package's own checkout (see below)	irrelevant

Neither affects what agents can do: an agent checks on background work over HTTP using the statusUrl handed to it in the reply, never a shell command. The CLI is for humans.

Or in any MCP client's config — "command": "npx", "args": ["-y", "@yanlinglabs/video-extract-mcp@latest"] for A, or "command": "video-extract-mcp" with no args for B:

{
  "mcpServers": {
    "video-extract": {
      "command": "npx",
      "args": ["-y", "@yanlinglabs/video-extract-mcp@latest"]
    }
  }
}

One gotcha with npx, and it only bites contributors. Run inside this package's own git checkout, npx @yanlinglabs/video-extract-mcp fails with command not found — npx sees the local package.json claiming that name, looks for the binary in a local node_modules/.bin that was never populated, and gives up. Since MCP clients launch servers with the working directory set to your project, option A cannot work in this repo. Working on the tool itself? Point that one project at your build — claude mcp add --scope local video-extract -- node "$PWD/dist/mcp.js" — which also means a npm run build takes effect immediately, with no publish round-trip. Everywhere else, npx is fine.

That is enough for any video that has captions — which, thanks to the caption-first transcript policy, is most of them. The vision model downloads itself on first use.

Speech models are only needed for videos with no captions at all, and they are fetched automatically the first time one is. Only the engine that video needs is downloaded — 233 MB for the Chinese/Japanese/Korean model, 1.3 GB for Whisper — into ~/.cache/video-extract-mcp/models. Set VIDEO_EXTRACT_AUTO_FETCH_MODELS=0 to keep it manual, or pre-fetch them yourself:

npx -y @yanlinglabs/video-extract-mcp --help   # installs the package
curl -fsSL https://raw.githubusercontent.com/yanlingLabs/video-extract-mcp/main/scripts/fetch-models.sh \
  | bash -s -- ~/.cache/video-extract-mcp/models

~/.cache/video-extract-mcp/models is where the tool looks by default. Override with VIDEO_EXTRACT_MODELS_DIR. If the fetch is disabled or fails, an uncaptioned video still returns frames and records a warning explaining why the transcript is missing — it degrades rather than fails.

From source (contributors)

git clone https://github.com/yanlingLabs/video-extract-mcp.git
cd video-extract-mcp
npm install && npm run build
./scripts/fetch-models.sh    # into ./models, which takes precedence when present
npm run preflight            # verifies ffmpeg / ffprobe / yt-dlp / tesseract

Environment variables

Variable	Purpose
VIDEO_EXTRACT_MODELS_DIR	Where speech models live. Defaults to ./models when that exists, else ~/.cache/video-extract-mcp/models.
VIDEO_EXTRACT_AUTO_FETCH_MODELS	Set 0 to stop the speech models being downloaded on demand. They are then your job (scripts/fetch-models.sh), and an uncaptioned video degrades with a warning saying so.
VIDEO_EXTRACT_COOKIES_FILE	Path to a Netscape-format cookie jar, used for every yt-dlp source at once — YouTube, Instagram, Facebook, X, TikTok, Twitch and the rest. See Authenticated sources.
VIDEO_EXTRACT_COOKIES_FROM_BROWSER	Load cookies from a local browser instead: chrome, firefox, safari, edge, brave, chromium, opera, vivaldi, whale, optionally browser:profile. Ignored when VIDEO_EXTRACT_COOKIES_FILE is set.
VIDEO_EXTRACT_WECHAT_COOKIE	A yuanbao session cookie, required only for WeChat Channels links. Separate from the above by design — a different protocol with its own credential.
VIDEO_EXTRACT_MAX_CONCURRENCY	Caps concurrent analyze_video item executions — plain calls and background tasks, batch items and separate calls, all count against the same limit. Default 4. resolve_video is exempt: it loads no models, so there is nothing to throttle.
VIDEO_EXTRACT_TASK_TTL_MS	How long a completed background-task handle stays queryable before it expires. Default 1800000 (30 minutes). 0 (or any non-positive value) means the handle never expires. Governs the in-memory handle only — files already written to destinationPath are never deleted by the tool, expired handle or not.
VIDEO_EXTRACT_STATUS_PORT	Pins the port of the localhost /status endpoint (see Watching progress). Unset picks an ephemeral port each start (default: endpoint on). The literal value 0 disables the endpoint entirely — note the contrast with VIDEO_EXTRACT_TASK_TTL_MS above, where 0 means no expiry, not disabled.

Authenticated sources

Plenty of media is not public, and the answer is the same one the platforms themselves ask for: cookies. One jar covers every yt-dlp source at once — cookies are scoped by domain inside the file, so a single export authenticates YouTube, Instagram, Facebook, X, TikTok and Twitch together. It is not a YouTube-only setting.

export VIDEO_EXTRACT_COOKIES_FROM_BROWSER=auto       # recommended: borrow only when blocked

auto is lazy, and that is the point. Ordinary requests send no cookies at all. Only when a platform actually refuses one does the server detect an installed browser, retry that single request with its cookies, and stop — no loop. You pay the cost of touching a credential store only when something is genuinely blocked, which is also what keeps a borrowed session from being rotated out from under you on every public video.

The other two modes are eager — cookies on every request:

Read a browser's store directly (chrome, safari, edge, brave, …):

export VIDEO_EXTRACT_COOKIES_FROM_BROWSER=firefox

Or point at an exported Netscape-format jar:

export VIDEO_EXTRACT_COOKIES_FILE=~/cookies.txt

Set both and the file wins — they are alternatives, not a pair.

Expect an OS keychain prompt. Every Chrome-family browser encrypts its cookie store against the system keyring, so the first read shows a dialog you must approve — on macOS, "Chrome Safe Storage". Firefox does not; its store is plain SQLite, which is why auto prefers it when both are present. If you set nothing at all and a request is refused, the reply tells you the command to enable this and warns about that prompt, rather than leaving you to discover it.

What it unlocks, beyond simply logging in: age-restricted and members-only YouTube, most of Instagram and Facebook, much of X, subscriber-only Twitch — and rate_limited / "sign in to confirm you're not a bot", which an anonymous fetch hits far sooner than a signed-in one.

Three things worth knowing before you use it:

• It is read from the environment only. A caller can never name a cookie file or a browser per-request. An agent that could do either could read any file on the machine, or lift a live session — so that stays the operator's decision, not the agent's.
• Your jar is never modified. --cookies FILE doesn't only read that file, it rewrites it on exit; the tool copies your jar to a private temp file, hands yt-dlp the copy, and deletes it afterwards. The cost is that refreshed cookies aren't written back.
• Exporting from a browser you are actively logged into can log you out. Platforms rotate session cookies, and a copy taken from a live session goes stale — YouTube is especially prone to it. yt-dlp's own advice: export from a private/incognito window, then close that window without logging out.

A wrong path fails loudly rather than quietly fetching anonymously — an unreadable jar is a broken setup, not something to degrade past, and silently anonymous results would send you hunting the wrong bug.

Check what you actually configured, without printing a single cookie value:

$ video-extract cookies
cookie jar: /Users/you/cookies.txt
  .youtube.com      12 cookies  expires in 23 days
  .instagram.com     4 cookies  expires in 3 days
  .x.com             2 cookies  EXPIRED 5 days ago
  3 domain(s), 18 cookies, 1 with expired cookies

It reports domains, counts and expiry — never names or values, so the output is safe to paste into an issue. It also names the mistake people actually make: an exporter set to JSON rather than the Netscape format produces a file that looks fine and contains nothing yt-dlp can read, which otherwise surfaces much later as a confusing auth_required on an unrelated video. Add --json for scripting.

Three ways to use it

The MCP server is the main surface, but the same engine is available two other ways.

As a CLI, which is the quickest way to see what it does before wiring up an agent:

npm run cli -- "https://youtube.com/watch?v=..." --max-frames 10 --out ./output

# just the transcript, no frames
npm run cli -- "<url>" --frames none --out ./output

# one exact frame at 7s, as cheap as this gets
npm run cli -- "<url>" --start 7 --end 7 --frames even --max-frames 1 --no-transcript --out ./output

It writes manifest.json plus the frame images into --out, and also prints the manifest to stdout.

If you want to pipe that JSON somewhere, call the built entry point directly — npm run prefixes its own banner lines to stdout, so npm run cli output is not valid JSON on its own:

npm run build
node dist/cli.js "<url>" --max-frames 10 | jq '.transcript.source'

As a library, if you want the pipeline without an agent in the loop:

import { analyzeVideo } from '@yanlinglabs/video-extract-mcp/dist/analyze.js';

const manifest = await analyzeVideo('https://youtube.com/watch?v=...', {
  start: 30, end: 90, frames: 'key', maxFrames: 12, outDir: './output',
});
console.log(manifest.transcript?.source);   // 'manual' | 'auto' | 'asr'
console.log(manifest.frames.map((f) => f.image));

analyzeVideo never throws for expected failures — a DRM page or a dead link comes back as a manifest whose source.status is not 'ok', carrying a readable reason. Check processing.warnings too: any optional stage that failed and was skipped past records an entry there.

Note that both the CLI and library paths run the compiled output. The speech and vision models run in separate worker processes resolved next to the compiled module, so running the TypeScript sources directly leaves those workers unresolvable — they degrade to a warning rather than an error, which is quiet enough to miss. npm run cli builds first for this reason.

The tools

Two that do the work, plus one lookup.

The surface is deliberately small. Earlier versions had four tools and the descriptions had to shout about which ones took URLs versus local paths — a sign the design was wrong, not that the warning needed to be louder.

resolve_video — look it up, optionally fetch it

resolve_video({
  destinationPath: string,          // required — shared by every item below
  videos: [{                        // one entry per video, at least one
    url:             string,        // required
    returnVideo?:    boolean,       // default false: metadata only, no download
    start?:          number,        // seconds; only with returnVideo: true
    end?:            number,
    comments?:       boolean,       // default false — slow on popular videos
  }],
})

One video — the common case, written flat into destinationPath:

resolve_video({
  destinationPath: "./out",
  videos: [{ url: "https://youtube.com/watch?v=..." }],
})
// -> ./out/metadata.json

Several videos in one call — each gets its own subdirectory, video-1/, video-2/, ... in array order:

resolve_video({
  destinationPath: "./out",
  videos: [
    { url: "https://youtube.com/watch?v=..." },
    { url: "https://tiktok.com/@user/video/...", returnVideo: true },
  ],
})
// -> ./out/video-1/metadata.json
// -> ./out/video-2/metadata.json + source.mp4 (returnVideo: true)

By default it downloads nothing heavy. You get title, creator, duration, the chapter list when the platform publishes one, and a short description preview. That is usually enough to decide what to do next — and it composes with ranges into the workflow that makes this whole thing efficient:

Read the chapters → see the demo starts at 12:04 → analyze only 12:04–20:00 → skip 90% of the download, transcription, and frame work.

analyze_video — the real work

analyze_video({
  destinationPath: string,                      // required — shared by every item below
  videos: [{                                     // one entry per video, at least one
    pathOrUrl:       string,                     // URL *or* a local file — both work
    start?:          number,                     // seconds
    end?:            number,                     // end === start means one instant
    frames?:         "key" | "even" | "none",    // default "key"
    maxFrames?:      number,                     // default 35
    transcript?:     boolean,                    // default true
    language?:       string,                     // optional override, e.g. "zh"
  }],
})

One video — the common case, written flat into destinationPath:

analyze_video({
  destinationPath: "./out",
  videos: [{ pathOrUrl: "https://youtube.com/watch?v=..." }],
})
// -> ./out/manifest.json, ./out/transcript.json, frame images

Several videos in one call — each gets its own subdirectory, video-1/, video-2/, ... in array order, and one item failing never fails the others:

analyze_video({
  destinationPath: "./out",
  videos: [
    { pathOrUrl: "https://youtube.com/watch?v=...", maxFrames: 10 },
    { pathOrUrl: "./local-clip.mp4", frames: "none" },
  ],
})
// -> ./out/video-1/manifest.json, transcript.json, frame images
// -> ./out/video-2/manifest.json, transcript.json, no frame images (frames: "none")

• "key" runs the importance selector and returns the best frames, deduplicated.
• "even" samples the range uniformly — maxFrames sets the density, so 60 frames across 30 seconds is 2fps.
• "none" returns no frames at all. That is how you ask for a transcript alone.
• One exact frame: start: 7, end: 7, frames: "even", maxFrames: 1, transcript: false.

Frame selection is bounded to start–end in both modes, and the transcript covers only the selected range.

get_status — collect a result after a timeout

get_status({
  videos: [string, ...]        // required — the same URLs or paths you passed before
})

Returns one entry per video: finished with the result the original call would have given you, running with the stages it has reached, or unknown. Only reach for it when a call actually timed out — see below.

Recovering a call your client stopped waiting for

Some clients cap how long they will wait for a tool call. When that happens the work carries on — the server keeps going and writes its files; only the client stops listening. To collect the result afterwards, ask about the video:

get_status({ videos: ["https://youtube.com/watch?v=..."] })

Pass the same URL or file path you gave analyze_video or resolve_video, several at once if you like. A finished video returns exactly the result the original call would have given you; one still running returns the stages it has reached.

Nothing has to be prepared in advance — that is the point. The URL is required on every call, so you always have it, and recovery matters precisely when nobody thought to set it up. Matching is exact: a ?si= parameter added or removed is a different string and answers unknown, because guessing which video you meant is worse than saying it does not know.

Records live in the server's memory and expire with VIDEO_EXTRACT_TASK_TTL_MS, so a restarted server answers unknown too — and says so plainly, because the files at destinationPath are the durable result either way.

Background tasks

Both tools are task-capable. Called as a plain MCP tool call, every example above behaves exactly as shown, on every client, whether or not it knows what a task is — with a small latency floor of about 150ms, which is invisible next to a real download but not zero on a cheap metadata-only call. Called as a task — an MCP client marks the call that way, using the (experimental) MCP tasks capability — the tool returns a handle immediately instead of blocking, and pushes progress while the work runs. This matters most for analyze_video, where a real video can take minutes.