sonicmatch-mcp

Your agent picks a song that doesn't fight the voiceover.

Drop footage. Get a shortlist that matches the picture, a 12–20s hook, and an ffmpeg ducking spec — with the license printed on every row.

Source: js713-lab/sonic-match-mcp. The installable package and CLI are named sonicmatch-mcp.

Video-to-BGM already exists. The wedge is not “I also match music”:

• it watches the footage, not the script
• it returns a hook window + ffmpeg ducking spec
• it is agent-native
• it prints the license instead of lying

Catalog quality will kill or save this. More tools will not.

Video or URL in
  → scene / mood / pace / speech analysis
  → license-safe BGM shortlist
  + beat/cut hints
  + optional mix preview

Do not treat this as “script in → YouTube Music search out.” That already exists (mcp-bgm-recommender). Sonicmatch watches the video.

You own	You do not own
Local file / public URL ingest	Platform music licenses
Mood, energy curve, speech vs silence, scene cuts	Meta/TikTok “trending audio” graph
CC / royalty-free catalogs + optional paid adapters	Spotify / IG official libraries
Ranked tracks, preview URLs, mix spec, ffmpeg	Auto-publish to Instagram

North star: ingest_video → analyze_video_music → recommend_bgm → preview_mix → export_mix_spec

License warning (read this)

• The code is MIT.
• Every track has its own license. It is printed on every recommendation.
• Nothing here is an official Instagram sticker, TikTok Commercial Music Library track, or YouTube Audio Library API result.
• Do not recommend commercial pop unless the adapter is explicitly a user-owned licensed library.
• CC-BY still needs attribution. CC-BY-NC is not ok for ads / shops. Non-commercial tracks are never auto-recommended.
• For ads / shops, wire a user-owned Artlist / Epidemic JSON (examples/user_library.example.json). Do not scrape those sites.
• Content ID can still hit you if you point at the wrong source. A CC label is not a waiver.

Quick start

Requires Python 3.10+ and ffmpeg / ffprobe on PATH. yt-dlp is optional and off by default (SONICMATCH_ALLOW_YTDLP=0) because platform extractors break and may violate ToS. Prefer a local file.

pip install git+https://github.com/js713-lab/sonic-match-mcp.git

# or from a clone
git clone https://github.com/js713-lab/sonic-match-mcp.git
cd sonic-match-mcp
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
cp .env.example .env   # optional keys

# stdio (Claude Desktop / Cursor)
sonicmatch-mcp

# streamable HTTP (web editors)
sonicmatch-mcp --http --port 8765

With uv:

uv venv && uv pip install -e ".[dev]"
uv run sonicmatch-mcp

v0.2 works offline-ish with a 20-track seed catalog aimed at Reel editors (cafe, product, talking-head, travel, food, fashion, event). Gemini, Jamendo, and Freesound are optional and degrade with a note in the tool response. Seed rows have no hosted audio on purpose — preview_mix synthesizes a demo bed. For real ads, point SONICMATCH_LIBRARY_PATH / EPIDEMIC_LIBRARY_PATH / ARTLIST_LIBRARY_PATH at JSON you already licensed.

# tests (generates tiny color mp4s with ffmpeg)
pytest

Example agent prompt

I dropped ./clip.mp4. Analyze it for an Instagram Reel and recommend 5 instrumental BGMs. Then mix the top pick with ducking and give me the ffmpeg command.

Claude Desktop

claude_desktop_config.json:

{
  "mcpServers": {
    "sonicmatch": {
      "command": "/absolute/path/to/sonicmatch-mcp/.venv/bin/sonicmatch-mcp",
      "args": [],
      "env": {
        "GEMINI_API_KEY": "",
        "JAMENDO_CLIENT_ID": "",
        "FREESOUND_API_KEY": ""
      }
    }
  }
}

Cursor

.cursor/mcp.json (project) or ~/.cursor/mcp.json:

{
  "mcpServers": {
    "sonicmatch": {
      "command": "uv",
      "args": ["--directory", "/absolute/path/to/sonicmatch-mcp", "run", "sonicmatch-mcp"]
    }
  }
}

Copy-paste configs live in examples/claude_desktop.mcp.json and examples/cursor.mcp.json. User-owned Epidemic/Artlist JSON shape: examples/user_library.example.json. Registry metadata: server.json.

HTTP editors can point at http://127.0.0.1:8765/mcp after sonicmatch-mcp --http.

--http has no authentication. Keep it on loopback. The Docker image binds 0.0.0.0 so the container port works — do not publish that port to the internet. See SECURITY.md.

Architecture

flowchart TB
  subgraph mcp [MCP Server - FastMCP / Python - stdio + HTTP]
    tools[ingest_video / analyze_video_music / recommend_bgm / preview_mix / export_mix_spec / suggest_cuts]
  end
  tools --> ingest
  tools --> brain
  tools --> hub
  tools --> mixer
  ingest[Ingestor<br/>yt-dlp · ffmpeg · ffprobe · URL/file]
  brain[Video Brain<br/>Gemini / local VL · librosa · PySceneDetect · Whisper]
  hub[Music Hub<br/>seed CC · Jamendo · Freesound · user library · generate]
  mixer[Mixer<br/>ffmpeg · ducking · loop/trim · EDL cuts]
  hub --> index[Track index<br/>tags + license + embeddings · SQLite · optional LanceDB]

Hard rule: never send raw multi-MB video through the MCP payload. Store locally, pass an asset_id. Loopback, file://, and private IPs are rejected (SSRF).

MCP tools

Tool	Input	Output
status	—	ffmpeg / keys / seed count
ingest_video	local path or HTTPS URL, max_seconds=180	asset_id, duration, probe, keyframe paths. Platform URLs need SONICMATCH_ALLOW_YTDLP=1
analyze_video_music	asset_id + platform + notes	VideoSonic profile
recommend_bgm	profile or asset_id + prefs + brand_kit	3–7 ranked tracks + reasons + license + hook in/out
search_music	free text / bpm / mood	catalog hits
get_track	id	metadata + license + urls
preview_mix	asset_id + track_id + ducking	preview files + ffmpeg recipe + mix spec
export_mix_spec	asset_id + track_id + render?	mix spec + ffmpeg + attribution (no render unless asked)
suggest_cuts	asset_id + optional bpm/track	beat grid, snapped scene cuts, EDL, intro/peak/outro
generate_bed	prompt / bpm / duration + i_understand_not_commercially_cleared=true	source=generated track (not catalog-cleared; excluded from auto recs)
save_brand_kit	BPM / moods / no-vocals	persisted kit name for recommend_bgm(brand_kit=…)
analyze_batch	list of paths/URLs (max 20)	mood cluster + shared mini-playlist

Also ships a prompt template: “Score this video like an IG music sticker.”

Product rules (Instagram-like, not Instagram)

• Prefer instrumental when speech_coverage > 0.25
• Recommend a hook window, not the whole song
• Show why (cuts at 0.8s average, 112 BPM, warm gold hour)
• Always return license + attribution text
• 3–7 tracks, not 40
• User can override mood / genre / no-lyrics / platform / energy
• Never claim “cleared for Instagram official sticker” unless it actually is

VideoSonic profile

Analysis returns structured JSON, not a paragraph:

{
  "duration_sec": 18.4,
  "aspect": "9:16",
  "content_type": "lifestyle",
  "has_speech": true,
  "speech_coverage": 0.62,
  "existing_music": false,
  "overall_mood": ["warm", "playful"],
  "energy_mean": 0.62,
  "energy_curve": [{"t": 0, "energy": 0.3}, {"t": 4, "energy": 0.8}],
  "pacing": "fast-cut",
  "scenes": [{"start": 0, "end": 3.2, "description": "cafe exterior", "energy": 0.4}],
  "hook_window": [9.0, 15.0],
  "suggested_bpm": [95, 118],
  "avoid": ["dark cinematic drone", "aggressive trap", "lyrics-dense"],
  "search_queries": ["warm acoustic pop instrumental cafe"],
  "platform_hint": "instagram_reel",
  "analyzer": "local"
}

• Primary: Gemini video understanding when GEMINI_API_KEY is set.
• Fallback: ffmpeg scene cuts + WAV energy / silence / ZCR heuristics. Optional faster-whisper, scenedetect, librosa if installed (pip install 'sonicmatch-mcp[local-vl]').

Music hub

Pluggable, license-first. v0 ships:

Adapter	When	License reality
Seed catalog (data/seed_tracks.json)	always	20 CC0 / CC-BY Reel beds + a vocal fixture + a CC-BY-NC fixture (NC is never auto-recommended)
Jamendo	JAMENDO_CLIENT_ID	CC, check commercial
Freesound	FREESOUND_API_KEY	CC, good for beds/loops not songs
User library JSON	SONICMATCH_LIBRARY_PATH / EPIDEMIC_LIBRARY_PATH / ARTLIST_LIBRARY_PATH	you already licensed it; we do not scrape paid sites
Generate	generate_bed	always source=generated; local sine demo unless you swap a real model

Ranking (weighted): mood/energy → instrumental if speech → duration/loop → BPM vs cut rate → license fit → tag embedding cosine → user constraints. recommend_bgm drops non-commercial and generated tracks instead of downranking them.

Tracks are indexed in SQLite (~/.cache/sonicmatch-mcp/db/tracks.sqlite) with a 24-d tag embedding. If lancedb is installed (pip install 'sonicmatch-mcp[embeddings]'), vectors are also upserted there.

Seed tracks have no remote audio files on purpose (you should host files you actually have the rights to). preview_mix synthesizes a CC0 demo bed so the mixer still runs offline. generate_bed is a catalog-miss fallback and is not cleared for ads.

Docker

docker build -t sonicmatch-mcp .
# Loopback-only publish. The process inside the container has no HTTP auth.
docker run --rm -p 127.0.0.1:8765:8765 -v sonic-cache:/data/cache sonicmatch-mcp

Roadmap

Catalog > new tools.

• Freesound adapter (loops / beds)
• Tag embeddings in SQLite (+ optional LanceDB extra)
• Epidemic Sound / Artlist as user-owned JSON plugins (no scrape)
• Beat-grid vs scene-cut suggestions (EDL-ish suggest_cuts)
• MCP registry listing (server.json)
• Generate tool, marked source=generated (local demo; swap a real model at your own legal risk)
• Official MCP registry listing via GitHub Release MCPB (see PUBLISH.md)
• Non-commercial licenses excluded from auto recommend_bgm
• 20 seed beds a Reel editor would actually keep, with audio you host
• User-owned Artlist / Epidemic JSON as the default path for ads
• Real CLAP audio embeddings
• PyPI release

Why this can be a good open-source project

Yes if you nail: (1) video-native analysis, (2) license honesty on every row, (3) editor-shaped output (hook in/out, ducking, mix spec), (4) a catalog someone would keep.

No if you only wrap YouTube Music search, or if the first five recs sound like leftover stock beds.

Day-1 risk gates (enforced in code, not slogans):

Risk	Gate
Content ID	Every rec/search/get_track includes content_id_warning. CC/RF is never "Content-ID-safe". content_id_risk is unknown or likely, never cleared.
yt-dlp ToS / broken extractors	Platform URL ingest is off unless SONICMATCH_ALLOW_YTDLP=1. Failures map to YTDLP_EXTRACTOR and tell you to pass a local file.
Upload size / SSRF	HTTPS-only remote ingest, no file:// / loopback / private IPs, SONICMATCH_MAX_DOWNLOAD_MB (default 200) on files, HTTP, and yt-dlp --max-filesize.
“Trending” is a closed Meta graph	Queries for trending/viral/IG audio/TikTok sound return empty + TRENDING_UNAVAILABLE. recommend_bgm always sets trending_available=false.
Generation-model commercial terms	generate_bed refuses unless i_understand_not_commercially_cleared=true. Generated tracks are excluded from auto recommend_bgm.

Use cases

IG Reel / Story · Shopee product clip · YouTube Shorts agent · CapCut/Premiere companion · campus recap · podcast clipper · travel-vlog batch · brand-kit lock (BPM + no vocals) · silent-film / accessibility · multi-agent studio.

License

MIT. Track licenses are independent of the repo license. Security reports: SECURITY.md.