MM3 LM Adapter Training — the working recipe

Goal this recipe is tuned for: an album clone. You want a generation that could be another track off that record. Memorisation is acceptable and in fact desirable; style bleed is acceptable. This is NOT tuned for a surgical trigger LoRA that leaves the base model otherwise untouched.

Established over a 6-album, 30,000-step sweep on 2026-08-23/24 (album B, the album-C artist, album K, album M, album L, the album-A artist), each album laddered by ear across 20 checkpoints. Working log: docs/plans/ 2026-08-23-mm3-albumC-style-adapter-findings.md (gitignored, local only).

THE CURRENT BEST RECIPE

Keep this block up to date — it is the thing the in-app trainer should eventually be dialled to, and docs/plans/ is gitignored so nothing there survives a fresh clone.

WHOLE-SONG since 2026-09-11. The crop-750 / history-2048 recipe below this block's history taught adapters that lost the song's arc: one sung passage, minutes of looped instrumental, no ending (vocal share 0.16 on album P against the album's 0.68). Nothing in a 30 s window ever scores more than 30 s at once. On the 2026-09-10 overnight those adapters ended 12 of 72 renders across six fresh albums while the pristine base ended 36 of 36 on the same prompts, with no dependence on album length or genre. Training every track as ONE sequence (--max-frames 9000, no prefix, flash) ended 7/12 on albums B and S and 6/12 on album P at the 360 s ceiling, and lifted album P's vocal share to 0.42. Cost on an RTX 5090: 6-12 s/step, 29 GB peak at rank 128 (the engine clamps its buffers to the album's longest track since 2026-09-11, so short albums cost less). Whole-song ending times follow the LYRIC SHEET's length more than the album's. Not yet re-heard for likeness at the time of writing; the ledger docs/plans/mm3-endings-checklist.md has the tables.

--lm mm3-lm-q8_0.gguf
--rank 128 --alpha 128 --adapter-type lora --hot-pizza          # HOT-PiZZA: PiSSA with principal-subspace dropout (2026-09-06)
--pissa-frozen-f16 --pissa-cache-dir <adapters>/mm3-lm-adapters/_pissa-init-cache
--optimizer adamw --lr 8e-5 --lr-end-frac 0.005 --warmup 25    # AdamW tied Prodigy by ear, 2.3 GB lighter (2x LR broke vocals)
--attn flash                                                   # no --prefix-frames: under the whole-song window the track is its own history
--max-frames 9000 --drop-over-frames 9000                      # WHOLE SONG (2026-09-11, Rob): every track as one sequence; tracks over 360 s left out
--crop-mode structured --crop-start-frac 0.2 --crop-end-frac 0.15   # only matters for a track longer than the window (none, with the drop)
--crop-start-tiles 3 --crop-anchor song
--rank-dropout 0.1                                             # the mask IS the method under --hot-pizza; never 0
--steps 600 --save-every 100                                   # Balanced = 600, Fast = 300, Thorough = 900 (2026-09-11); stop on STEPS
--depth-loss-weight 1.0 --depth-loss-frames 128
# captions: per-track <stem>.mm3.txt from MOSS/Gemini, and ONLY those. No --caption-file:
# the shared caption killed endings (0/6 vs 4/6) and was removed on 2026-09-09.
# NO --reg-* prior by default (it was a workaround for the shared caption; costs likeness)
--trigger "<artist>" --trigger-prepend
--holdout 0.15 --eval-every 250 --eval-crop 500

Previews: every 50 steps (= every checkpoint), 40 s, control + baseline off, rendered on q8_0 at MLP 1.0 — the same dials generation uses.

Rob, 2026-09-06, on the stacked HOT-PiZZA recipe above (blind-hotpissa-recipe letter F): "sounds fantastic, this should be the default in-app." Cost on albumA: 4.7 s/step, 39 min for 500 steps, peak 25.9 GB — against the 2026-09-05 LoKr/Prodigy/exact/4096 line's 5.9 s, 49 min, 30.4 GB. Every lever was first tied individually in a blind set, then stacked and heard. The earlier LoKr line stays in the git history of this file.

2026-09-07, the safe stack. An overnight one-lever-at-a-time speed trial (overnight-speed/COSTS.md + blind-speed/RESULTS.md in the album A hub) found prefix 1024 (69 of 90) and crop 500 (67.5) tie the reference (68), while the prefill chunk 256 → 1024 cuts 17% off the step with an identical step-1 loss. Combined and heard blind (blind-confirm/RESULTS.md): safe stack 67 vs the crop-750 recipe 70.5, inside the ~6 noise floor, at 3.1 s/step and 26 min per 500 steps (was 4.5 s, 37 min). That is the recipe block above. What did NOT survive: doubling the LR (every 2x arm 3-4 under, and a 2x-LR/300-step stack produced a vocal-free plan on 1 of 6 songs across two seeds) and turning the acoustic loss off (lowest score). Held-out loss every 50 has the same shape in every arm — it follows the seed's crop order, not the adapter — so it is not a stopping signal.

Presets (Rob, 2026-09-07, MM3_LM_PRESETS in mm3Train.ts, a Recipe row in the Training Studio card; the route lays preset under the request's own fields, so {preset:'thorough'} alone trains Thorough):

preset	steps	lr	window	history	min/album (5090)	record
Fast	300	8e-5	whole song (9000)	none	30-60	the 2026-09-10 overnight arm: album B 7/12, album S 7/12, album P 3/12 at 300 s and 6/12 at 360 s
Balanced (default)	600	8e-5	whole song (9000)	none	60-120	Fast at twice the depth; Rob's pick for the default (2026-09-11), not yet heard
Thorough	900	8e-5	whole song (9000)	none	90-180	three times Fast's depth; not yet heard

(Superseded 2026-09-11: Fast 300 / Balanced 500 / Thorough 1000 at crop 750 with a 2048- or 4096-frame history. That regime's record — Fast's Simlish vocals of 2026-09-07, Balanced's 4/6 on album B — is in this file's git history.) All three share flash, AdamW, HOT-PiZZA r128, f16 factors and the acoustic loss; tracks longer than the window are left out (longTracks: 'exclude', engine --drop-over-frames). MM3_LM_DEFAULTS carries the Balanced values, so an empty API body trains Balanced. Prior preservation is OFF unless the request names a corpus (since 2026-09-09).

Rob, 2026-08-25, on the LoKr configuration this replaced: "the closest we've ever gotten to artist replication." Crop 750 = 30 s = ~3 s/step; he set it by ear after finding 15 s steps at crop 4272 unworkable and the shorter crop better, not merely faster.

At render: everything 1.0 for adapters trained WITH the acoustic loss (2026-08-25 onward) — previews and generation now default there. The old "MLP 0.63-0.75" dial was damage control for the timbre fault and applies only to PRE-FIX adapters (their sidecar recommendedScales override the defaults).

19.7 GB VRAM measured at crop 750 (the acoustic loss adds its frozen depth decoder, ~1.2 GB). The crop settings changed on 2026-08-24 and everything auditioned before that date was trained at --max-frames 128 --crop-mode random — five seconds per step. Treat pre-2026-08-24 ear results about the BASE and the OPTIMIZER as void (see below); the rank and MLP-dial findings stand, because they were measured against each other under the same broken crop.

The crop is the axis that decides whether it sounds like a song

Found 2026-08-24 after every checkpoint of an album Q run rendered a track that began part-way through a song and faded out mid-render without resolving.

The crop was 128 frames — 5.12 seconds — against a 204 s median track. Two separate defects came out of that, and the second is the one that bites:

1. Under-coverage. Measured on the reference dataset (10 tracks, median 5099 frames):

crops reaching the track end, the ONLY place EOS is supervised	2.6%
crops starting at frame 0	0.02%

So the model was asked at render time to produce an opening and an ending it had essentially never been shown.

2. Random crops teach the wrong lesson. A random crop presents the prompt followed immediately by mid-song audio with no history in front of it, and supervises it. Every one of those teaches "a song may legitimately begin at position c0". A render that starts mid-flow, or stops and restarts, is the model doing what it was trained to do. This framing is ScragBot's and it is better than "under-coverage" — it explains the fades, which under-coverage alone does not.

Hence --crop-mode structured: 85% of steps anchored at frame 0, so every supervised position carries the song's real history exactly as it will at generation time, and 15% flush to the track end, which is the only place EOS is supervised. beginning cannot do the second and random cannot do either.

Whole-track training is not reachable on a 32 GB card. 9 of 10 reference tracks exceed 4096 frames and the median would need ~17 GB of attention scores alone.

Crop length is QUADRATIC in VRAM, and that is why f16 lost

Peak is loaded + perRank*rank + 0.2679*S + 0.00044765*S^2 + const, with S = 1142 + frames. The backward retains [S, S, heads] attention scores — that was true unconditionally until 2026-09-05. ggml's flash-attn has no backward is no longer correct: HOT-Step's own fused GGML_OP_FLASH_ATTN_TRAIN/_BACK closed that gap on 2026-09-01, and mm3-lm-train gained --attn flash|flash-f32 on 2026-09-05, making attention memory linear in S instead of quadratic. Measured on this corpus (RTX 5090, mm3-lm-f16/mm3-lm-q8_0, rank 256, checkpointed): flash moves the usable crop ceiling from ~4300 frames (exact, table below) to at least 11,178 frames — this dataset's longest track, no OOM reached — before the same ~29 GB practical spill ceiling. Default is still exact, the table below is unchanged, and none of this is in the recipe — the shipped recipe trains at crop 750, where flash measures no benefit (checkpointed segments already hide the small softmax in allocator slack), and nothing trained under flash has been heard. Full numbers: docs/TRAINING.md MM3 section, .claude/skills/flash-attn-training/SKILL.md §3/§7. What fits in 30 GB at --attn exact, the only mode any shipped adapter has trained under:

config	max crop	covers a 204 s track
f16 r128 prodigy	1650 fr / 66 s	32%
f16 r128 adamw	2496 fr / 100 s	49%
f16 r64 adamw	3190 fr / 128 s	63%
q8_0 r128 adamw	4272 fr / 171 s	84%
q8_0 r64 adamw	4771 fr / 191 s	94%

f16 and Prodigy together were holding 10.6 GB, which is ~2600 frames of crop. Both were dropped for the crop.

This reverses the f16-over-q8_0 ear result deliberately. That test (a 750-step f16-trained adapter beating a 2000-step q8_0-trained one) was run at the 128-frame crop, where both candidates had been trained on five-second fragments and neither had learned how a song starts or ends. It compared two structurally broken adapters. Re-run it at this crop before spending 8 GB on f16 again.

BF16 tensor cores (--weights bf16) — works, but not on a 32 GB card

The trainer runs base matmuls in F32: ggml_out_prod is F32-only, so lm_linear dequantizes each weight in-graph and the GEMMs land on TF32. Lever A (engine/src/train/lm-bf16.h) feeds the raw BF16 weight to mul_mat and rewrites the backward's OUT_PROD nodes into MUL_MAT, reaching the tensor cores. Wired into mm3-lm-train on 2026-08-24; needs a BF16 base from convert-mm3.py --components lm --quant bf16.

Measured on a 5090, matched in every other respect:

config	crop	track	step	ms/frame	peak	free
q8_0 + F32 window	4272	84%	15.50 s	3.63	27.7 GB	4.2
q8_0 + F32 window	2496	49%	10.50 s	4.21	20.1 GB	11.8
bf16 + Lever A	2496	49%	7.50 s	3.00	28.9 GB	3.0
bf16 + Lever A	3100	61%	9.75 s	3.15	30.6 GB	1.2

1.4x faster at matched crop, for 9.0 GB — 7.7 GB of which is simply the base being 16-bit rather than q8_0. That ceilings bf16 at ~3100 frames where q8_0 reaches 4272, so the speed costs 23 points of track coverage, and coverage is what decides whether a render sounds like a song. Per supervised frame bf16 is only 1.15x ahead there.

Default stays q8_0. Pick bf16 when coverage is not the binding constraint: a bigger card, a shorter corpus, or a deliberate speed run. Since 2026-09-03 the base is downloadable: mm3-lm-bf16 is on scragnog/MiniMax-Music3-GGUF and in the registry, and picking it in the train form's base picker is all it takes (the server maps a bf16 base to --weights bf16). Untested on Ampere (A40), where the F32 fallback is slowest and the gain should be largest.

One loose thread: identical step-1 loss (3.5930 vs 3.5932) at a 29% lower gradient norm (5.561 vs 7.870) is the quantizer's error appearing as gradient noise. Whether that matters by ear is untested.

BF16 is also the SOURCE dtype of the MM3 weights — but it is not better for inference than f16, which keeps all 7 of BF16's mantissa bits and adds 3 more. Render on q8_0 as always.

HOT-PiZZA is the default method since 2026-09-06 (Rob)

--hot-pizza (implies --pissa): PiSSA with the rank-dropout mask on the principal component itself, so every micro-step a random --rank-dropout share of the base's top-128 subspace is deleted while the adapter fits the album. Found as a masking bug on 2026-09-05 (lm-graph.h masked one branch of the PiSSA fold), kept on purpose: blind on albumA it beat every other method twice (68 and 66.5 of 90; LoRA 60–64, DoRA 63, LoKr 41.5 with a drone failure; the CORRECTED PiSSA was worst at 39.5 with drone plans in half its renders). Tables: _experiments/_LISTENING/2026-09-06-mm3-method-ab/albumA/. Consequences for the recipe block above: --adapter-type lora --rank 128 --alpha 128 --hot-pizza replaces the LoKr line; stop on steps (500 heard; 250/350 ladder pending), never on loss — the perturbed forward keeps the trailing train loss at 2.7–5, so a loss target never binds. Plain LoRA is next in line; LoKr stays selectable. One album so far: the second-artist run is the outstanding validation. Cost is LoRA's: ~5.9 s/step, peak 30.4 GB on the 32 GB card at crop 750 + prefix 4096.

Endings, honest status (2026-09-10)

Tool: tools/vocal-end/vocal_end.py (SuperSep vocal stem + energy trace + whisper hint) tells you where singing starts and stops in any render or dataset track; use it before believing an LRC timestamp or guessing at a tail.

Per-track captions are necessary (the shared caption gave 0/6) but not sufficient: the same recipe's per-plan natural-ending rate is 0.2-0.5, varies with the caption x lyrics pair and with the training draw (the trainer is deterministic per seed; two draws gave 2/6 and 3/6 with equal likeness), and Rob's in-app test on three fresh albums was 2 of 9. No training-side lever has raised it without losing likeness (score-last ends everything and sounds like nothing). The natural-ending candidates feature (3 takes, up to 4 rounds) is what makes a render end. The full ledger and the ordered list of what is left to try: docs/plans/mm3-endings-checklist.md. Pitch/tempo drift in adapter renders (every arm with the acoustic loss; DL0 clean) has its own file: docs/plans/mm3-pitch-tempo-drift.md.

Adapter files: the residual + delta form (2026-09-09)

A HOT-PiZZA / PiSSA export used to be a rank-2r F32 PEFT LoRA: 2.79 GB per checkpoint at r128, half of it the frozen A0/B0 pair (the base weight's own top-128 singular directions), which is the SAME bytes in every adapter trained on the same base. Since 2026-09-09 that half ships once:

• models/mm3/mm3-lm-q8_0.pissa-r128.safetensors (0.7 GB, F16; registry id mm3-lm-q8_0-pissa-r128, in the Q8_0 / Balanced / Training packs; format in engine/src/pissa-residual.h). The trainer reads it at init when rank, SVD parameters and the base's byte size match, else falls back to the SVD cache or the SVD and WRITES it beside the base. Keyed by base file size: a re-quantized base misses rather than pairs wrongly.
• Adapters trained against it export the DELTA form: lora_A = A - A0, lora_B = s(B - B0) at rank r, F16, hot_step.param_method = 4, hot_step.pissa.meta, hot_step_pissa_residual in adapter_config.json. 0.7 GB per checkpoint instead of 2.8. minimax/mm3-lm-adapter.h rebuilds the rank-2r pair from the two files (needs the resident base's path, so the residual is found beside it); everything downstream is unchanged, VRAM too. NOT a plain LoRA: PEFT/SimpleTuner would apply (B-B0)(A-A0), which is nothing; the AS1.5 LM loader refuses the marker by name.
• Standalone rank-2r exports (no residual beside the base, or the AS1.5 LM trainer) are still written, now F16 (1.4 GB). Every older adapter loads as before.
• Missing residual at load: the engine error names the file and the Models page; download it there. Its SVD is deterministic but not bit-reproducible across GPUs, so the file is the truth, never recomputed on a user's machine for the shipped base.

Also new that day: saveEvery default 100 (was 50) and the ~4.2 GB resume-state.bin is deleted when a run reaches its end unless "Keep resume state after completion" is ticked (keepResumeState); a run that stops short keeps it. verifyExport: true on the request runs --verify-export (every checkpoint round-trips the runtime loader; used for the first delta run).

New adapter knobs (2026-09-04/05) — none of this is in the recipe above

mm3-lm-train gained the same six parameterizations train-dit has: --dora, --rslora, --hira, --loha, --pissa (+ --pissa-oversample, --pissa-iters) and --hra, mutually exclusive under the same rules as the DiT trainer (HiRA excludes DoRA; LoHa excludes both; PiSSA excludes DoRA/HiRA/LoHa; HRA excludes everything including rsLoRA and needs an even --rank) — plus two soft-prompt flags that now reach MM3 generation as well as training: --artist-token/--artist-token-k/--artist-token-lr and --prefix-n (a trained KV prefix, distinct from --prefix-frames above, which is frozen history with no gradient). All seven arms pass their finite-difference gate at the default epsilon since 2026-09-05, when --fd-eps became a floor on the step rather than the step (HiRA's gradients are 16-68x smaller than a plain LoRA's, so a fixed step measured it at 50x worse signal-to-noise — the estimator, not the backward). --hra is the one arm whose graph and activations scale with --rank: its default-rank (64) crash is fixed (graph budgets now come from the bank), but rank 64 still does not FIT at crop 750 on 32 GB and is refused with the arithmetic rather than crashing. Rank 8 peaks at 17.6 GiB and ~6.8 s/step, rank 32 at ~30.0 GiB and ~27 s/step, against a plain rank-64 LoRA's 18.2 GiB and ~2.4 s/step. --prefix-n and prior preservation (--reg-*) are mutually exclusive — a trained prefix is non-zero from init, and prior capture needs an inert model — and so are --hra/--rslora; the training routes 400 on both pairs rather than letting the job fail after the model loads.

None of this is in THE CURRENT BEST RECIPE block and none of it is ear-validated. The DiT trainer's own blind listening test found that no parameterization beat plain LoRA — treat that as the prior for MM3 too until MM3 has run its own test. Full gate numbers and the open HiRA/HRA bugs: docs/TRAINING.md MM3 section.

Supervising fewer positions does NOT buy VRAM

Worth writing down because it is an intuitive and wrong idea. Restricting the supervised span while keeping the visible prefix saves compute, not memory:

• peak is driven by the visible span S, not by s_tr;
• the CE head is already chunked — lm_ckpt_head_chunked loops for (i = 0; i < s_tr; i += CH) allocating per chunk from an arena sized by s_max, so the supervised count drives iteration count;
• allocation happens upfront at s_max regardless of the actual S per step.

The version that buys the memory is a no-grad frozen-KV prefix — condition on 0..E without retaining prefix activations, supervise only the tail. BUILT 2026-08-26, --prefix-frames N — see below.

--prefix-frames N: real history in front of the crop

Built 2026-08-26 (engine/src/train/lm-kvprefix.h; working notes in docs/plans/2026-08-26-lm-frozen-kv-prefix.md, gitignored). Off by default, never ear-tested, no UI — a run has to be launched by hand.

The problem it solves is not coverage, it is CONTEXT. A crop at frame 3000 carries its true RoPE position with ~750 keys of evidence in front of it, so the middle third of the stack — the layers doing long-range aggregation — is trained to produce position-3000 behaviour from a position-750 view. That is the band Rob switches off at render time with scaleMid 0.

A prefix needs no backward, so it escapes the quadratic term entirely: K and V cost 0.28125 MB per column across MM3's 36 layers. Measured at crop 750, rank 64, AdamW: +856 MB and about +60% step time for 750 frames (30 s) of history.

How much history. Measured over the corpus (202 tracks, 14 datasets, median 203 s), as the share of supervised steps whose crop sees as much history as it will at render:

prefix		full-context steps	mean prefix used	prefill cost
750	30 s	48.9%	505 fr	1.0x
2250	90 s	73.5%	1088 fr	2.8x
4096	164 s	87.7%	1453 fr	4.8x
5000	200 s	94.4%	1532 fr	5.4x
6000	240 s	98.0%	1566 fr	5.7x

4096 is the default: past it the curve flattens and you are spending quadratically to chase the tail. The flag is a CEILING, not a fixed cost -- a crop near the song's start has little history to load, which is why the mean at 4096 is only ~1450 frames. VRAM is linear and never the binding constraint (1.7 GB at 4096).

Two things it changes that are worth knowing:

• The window takes one extra input frame. With history present, the row that predicts frame c0 must be frame c0-1's, not the caption's last token. Without that shift, every crop still teaches "a song may begin at c0" — the exact lesson the crop work exists to remove — and the equivalence self-test catches it as a 0.083-nat gap.
• --prefix-selftest is the gate. Attention over [prefix ; window] is mathematically identical to one long crop covering both, so the supervised CE must not care which way it was produced. It found two real bugs before it passed. Run it before trusting a prefix run.

Do NOT set --crop-start-frac to 0 once a prefix is on

Tempting, because the start bucket's ORIGINAL justification was history: anchoring at frame 0 was the only way a supervised position got the song's real past in front of it, and the prefix now does that everywhere. But the bucket has a second job the prefix does not touch.

Frame 0 is the only place the caption-to-first-frame transition is trained. With a prefix, lead makes the previous FRAME the predictor of every supervised position -- correct for mid-song, and exactly what generation does after t=0. At c0 == 0 there is no previous frame, lead is 0, and the caption's last token is the predictor. That is the one case generation faces at t=0, and a uniform draw lands on it with probability 1/span, about 0.02%. Setting the share to 0 reintroduces the 2026-08-24 bug (renders that begin mid-flow) by a different route.

What the prefix DOES retire is the tiles. --crop-start-tiles 3 put half the start share on aligned tiles to teach the intro-build-verse arc, because a crop at 1500 otherwise had no past. It has one now, so a tile crop is just an ordinary crop. --crop-start-frac 0.20 --crop-start-tiles 1 is the reallocation: the opening keeps a share comparable to the ending's 0.15 (both are one event per song), and the freed 27% goes to random crops. Costs ~1.35x the prefill, because more steps land late where the prefix is longest. UNTESTED -- change it on its own run, not alongside a prefix change.