Audio System Architecture

This guide describes NullPlayer's audio playback system, including local file playback, streaming audio, equalization, spectrum analysis, and waveform generation.

Overview

NullPlayer uses two parallel audio pipelines to handle different content types:

Content Type	Pipeline	EQ Support	Spectrum	Waveform
Local files (.mp3, .flac, etc.)	AVAudioEngine	Yes	Yes	Cached 4096-bucket snapshot
HTTP streaming (Plex/Subsonic/Jellyfin/Emby/radio)	AudioStreaming library	Yes	Yes	Live stream accumulator from 576-sample PCM chunks

Both pipelines support the active EQ layout for the current UI mode and real-time spectrum visualization. Classic mode uses the legacy 10-band layout; modern mode uses a 21-band layout. EQ settings are automatically synchronized between them. Adaptive/dynamic spectrum modes share the same broad algorithm across local and streaming playback; accurate mode currently differs by pipeline (local uses BeSpec-style peak aggregation, streaming uses RMS power integration). The waveform window reuses the same playback sources: local files use cached snapshots, while streams start with live accumulation and may promote to a cached seekable snapshot when prerendering is available.

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────┐
│                         AudioEngine                                  │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  LOCAL FILES                        STREAMING (Plex/Subsonic)        │
│  ────────────                       ──────────────────────────       │
│                                                                      │
│  ┌──────────────┐                   ┌─────────────────────────────┐ │
│  │ AVAudioFile  │                   │   StreamingAudioPlayer      │ │
│  └──────┬───────┘                   │   (AudioStreaming lib)      │ │
│         │                           │                             │ │
│         ▼                           │  ┌───────────────────────┐  │ │
│  ┌──────────────┐                   │  │ HTTP URL → Decode →   │  │ │
│  │ playerNode   │─────────┐         │  │ PCM buffers           │  │ │
│  │ (primary)    │         │         │  └───────────┬───────────┘  │ │
│  └──────────────┘         │         │              │              │ │
│                           │         │              ▼              │ │
│  ┌──────────────┐         │         │  ┌───────────────────────┐  │ │
│  │crossfadeNode │─────────┼──► mixerNode ─► eqNode ────────────┐ │ │
│  │ (for Sweet   │         │         │  │ AVAudioUnitEQ         │  │ │
│  │  Fades)      │         │         │  └───────────┬───────────┘  │ │
│  └──────────────┘         │         │              │              │ │
│                           │         │              ▼              │ │
│                           │         │  ┌───────────────────────┐  │ │
│                           │         │  │ Spectrum Tap          │  │ │
│                           │         │  │ (frameFiltering)      │  │ │
│                           │         │  └───────────────────────┘  │ │
│                           ▼         └─────────────────────────────┘ │
│                    ┌──────────────┐                                 │
│                    │ mixerNode    │  EQ settings sync ◄──────────►  │
│                    └──────┬───────┘                                 │
│                           │                                         │
│                           ▼                                         │
│                    ┌──────────────┐                                 │
│                    │ eqNode       │                                 │
│                    │ (mode EQ)    │                                 │
│                    └──────┬───────┘                                 │
│                           │                                         │
│                           ▼                                         │
│                    ┌──────────────┐                                 │
│                    │mainMixerNode │                                 │
│                    └──────┬───────┘                                 │
│                           │                                         │
│                           ▼                                         │
│                    ┌──────────────┐                                 │
│                    │ Output Node  │ ─────► Speakers / Audio Device  │
│                    └──────────────┘                                 │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

Components Quick Reference

AudioEngine (Audio/AudioEngine.swift)

Main audio controller managing:

• Playback state (play, pause, stop, seek)
• Playlist management
• Shuffle cycle management for non-repeating playback order
• Track loading (routes to appropriate pipeline)
• EQ settings (synced to both pipelines)
• Gapless playback (optional; local files and same-pipeline streaming)
• Volume normalization (optional, local files only)
• Sweet Fades crossfade (both pipelines)
• Reference Tuning pitch shift (both pipelines via a shared PitchTuningController; disabled while casting)
• Playback Speed tempo control (0.25×...4.0×; local files and HTTP streams; disabled while casting)
• Output device selection
• Delegate notifications for UI updates
• Separate consumer gating for FFT/spectrum work vs live waveform chunk generation

Key Properties:

private let engine = AVAudioEngine()
private let playerNode = AVAudioPlayerNode()
private let crossfadePlayerNode = AVAudioPlayerNode()
private let activeEQConfiguration: EQConfiguration
private let eqNode: AVAudioUnitEQ
private var streamingPlayer: StreamingAudioPlayer?
private var crossfadeStreamingPlayer: StreamingAudioPlayer?
var gaplessPlaybackEnabled: Bool
var volumeNormalizationEnabled: Bool
var sweetFadeEnabled: Bool
var sweetFadeDuration: TimeInterval

Shuffle-specific behavior in AudioEngine:

• Shuffle playback uses a persistent cycle order instead of choosing a fresh random index for each advance.
• A shuffled cycle visits each playlist index once before stopping or reshuffling for repeat.
• Explicit track selection while shuffle is enabled re-anchors the cycle on that selected track.
• Queue replacement, playNow, and empty-queue insertion all start playback from the shuffled currentIndex, not from index 0.

StreamingAudioPlayer (Audio/StreamingAudioPlayer.swift)

Wrapper around AudioStreaming library:

• HTTP audio streaming with buffering
• Its own AVAudioUnitEQ (stays synchronized)
• Spectrum analysis via frame filtering
• Optional 576-sample waveform chunk generation for live waveform consumers
• State change callbacks

Why separate EQ? AVAudioNode instances can only be attached to one AVAudioEngine. Since AudioStreaming uses its own internal engine, we maintain a separate EQ node that stays synchronized with the main engine's EQ.

Track Switch Race Condition Guard

When switching streaming tracks, avoid race conditions with an isLoadingNewStreamingTrack flag:

private var isLoadingNewStreamingTrack: Bool = false

private func loadStreamingTrack(_ track: Track) {
    isLoadingNewStreamingTrack = true

    // DON'T call stop() before play() - AudioStreaming handles this internally
    streamingPlayer?.play(url: track.url)
    state = .playing

    DispatchQueue.main.asyncAfter(deadline: .now() + 0.05) { [weak self] in
        self?.isLoadingNewStreamingTrack = false
    }
}

func streamingPlayerDidFinishPlaying() {
    guard !isLoadingNewStreamingTrack else { return }
    trackDidFinish()
}

NAS Responsiveness for Local Track Switches

See skills/local-library/SKILL.md — NAS Responsiveness section.

Equalizer

Configuration

Classic mode uses the legacy 10-band configuration; modern mode uses a 21-band configuration derived from EQConfiguration.modern21. Both pipelines build their AVAudioUnitEQ from the same active layout at launch.

Classic 10-Band Configuration

Band	Frequency	Filter Type	Bandwidth
0	60 Hz	Low Shelf	1.0 octave
1	170 Hz	Parametric	1.75 octaves
2	310 Hz	Parametric	1.75 octaves
3	600 Hz	Parametric	1.75 octaves
4	1 kHz	Parametric	1.75 octaves
5	3 kHz	Parametric	1.75 octaves
6	6 kHz	Parametric	1.75 octaves
7	12 kHz	Parametric	1.75 octaves
8	14 kHz	Parametric	1.75 octaves
9	16 kHz	High Shelf	1.0 octave

Modern 21-Band Configuration

Frequencies: 31.5, 45, 63, 90, 125, 180, 250, 355, 500, 710, 1000, 1400, 2000, 2800, 4000, 5600, 8000, 11200, 14000, 16000, 20000

• First band: lowShelf

• Last band: highShelf

• Middle bands: parametric

• Parametric bandwidth: 1.0 octave

• Per-band gain: -12 dB to +12 dB

• Preamp (global gain): -12 dB to +12 dB

• Disabled by default to preserve original audio quality

• Saved EQ arrays are remapped between 10-band and 21-band layouts when restoring across classic/modern mode switches

Modern EQ UI Controls

Control	Behavior
ON toggle	Enable/disable EQ
AUTO toggle	Apply genre-based preset for current track; auto-enables EQ if off
FLAT ROCK POP ELEC HIP JAZZ CLSC buttons	Apply preset and highlight active button; auto-enables EQ if off; clicking active button deactivates (applies flat, no highlight)
Drag a fader	Adjust that EQ band; clears active preset highlight
Double-click a fader	Reset that band only to 0 dB (not all bands)
Integrated PRE control	Adjust global preamp from -12...+12 dB; double-click resets to 0 dB

Modern EQ specifics:

• 21 compact faders across the full slider strip
• Integrated glowing PRE control in the graph/header strip instead of a dedicated left preamp slider
• All 21 frequency labels are shown in-window using compact formatting (1K, 1.4K, 2K, etc.)
• Graph background uses per-band mini tracks so it visually echoes the fader lanes instead of a single connected fill

EQ Synchronization

When EQ settings change, both pipelines are updated:

func setEQBand(_ band: Int, gain: Float) {
    let clampedGain = max(-12, min(12, gain))
    eqNode.bands[band].gain = clampedGain      // Local pipeline
    streamingPlayer?.setEQBand(band, gain: clampedGain)  // Streaming pipeline
}

Reference Tuning and Playback Speed

Reference Tuning shifts playback pitch by a precise cents offset to retune content from one reference frequency to another (e.g. A=440 → A=432). It is implemented as a shared PitchTuningController (Audio/PitchTuningController.swift) that owns the local AVAudioUnitTimePitch node and creates one configured streaming pitch node per AudioStreaming player. This keeps the primary and Sweet Fades crossfade streaming graphs independent while driving all nodes from the same state.

Playback Speed is also owned by PitchTuningController, but it is a tempo/time-stretch control, not a pitch shift. The supported user range is 0.25×...4.0×; 1.0× is normal speed. Pitch is preserved while tempo changes.

Graph placement

Local graph: playerNode + crossfadePlayerNode → mixerNode → localPitchNode → eqNode → mainMixerNode. Placing the pitch node after the mixer means a single node handles both the primary and crossfade players, and the spectrum tap on mixerNode keeps capturing pre-pitch (source) frequencies — the analyzer shows the source content's spectrum, not the shifted output. This is a deliberate trade-off; moving the tap onto localPitchNode would invert it.

Streaming graph: each StreamingAudioPlayer receives its own node from PitchTuningController.makeStreamingPitchNode() and attaches it via AudioStreaming.AudioPlayer.attach(node:) after the EQ node. These streaming pitch nodes only carry Reference Tuning cents: they always keep node.rate = 1.0. Streaming tempo is applied through AudioStreaming's own private rateNode via StreamingAudioPlayer.rate / AudioPlayer.rate. This avoids double-applying rate when Reference Tuning and Playback Speed are used together.

When AudioEngine.setPlaybackSpeed(_:) changes the rate, it updates the controller, persists the preference, applies the rate to the active primary streaming player, and applies it to the crossfade streaming player if one exists. Newly-created streaming players must immediately inherit tuningController.rate.

Math and clamp

Cents offset = 1200 · log2(target / source) (e.g. 440 → 432 ≈ −31.766 cents). AVAudioUnitTimePitch.pitch is in cents, ±2400. The controller clamps appliedCents to that range; the Custom… dialog validates input at entry time and rejects out-of-range frequencies rather than silently clamping.

Playback Speed clamps via PitchTuningController.minRate / maxRate (0.25...4.0). Local file time display is wall-clock based, so local currentTime must multiply elapsed wall time by playbackSpeed; streaming time comes from AudioStreaming's own player progress.

Local time baseline (_currentTime + playbackStartDate)

For local playback the currentTime getter returns _currentTime + (now − playbackStartDate) · playbackSpeed while state == .playing (AudioEngine.swift, currentTime). _currentTime is a base offset re-established only at discrete events (load → 0, seek, pause, speed change); the 0.1s UI timer (startTimeUpdates) only reads currentTime, it never advances the baseline. The streaming and cast branches of the getter ignore playbackStartDate entirely.

Invariant: any code that resets playbackStartDate = Date() while already playing locally must first fold the elapsed time into the baseline, or currentTime collapses back to the stale _currentTime:

_currentTime = currentTime          // capture true position (old startDate still set)
lastReportedTime = _currentTime
playbackStartDate = Date()          // now safe to re-baseline

setPlaybackSpeed(_:) follows this idiom. play() also folds on a redundant press while playing (issue #348) but gates on state == .playing and the active local node still playing. The node clause is essential: the pipeline-reload path (commitLoadedLocalTrack) zeroes _currentTime and stops the node while leaving state == .playing, so folding there would read a stale playbackStartDate and jump the clock forward. The active node is crossfadePlayerNode when crossfadePlayerIsActive (e.g. after a completed Sweet Fades crossfade) and playerNode otherwise — check the right one (crossfadePlayerIsActive ? crossfadePlayerNode.isPlaying : playerNode.isPlaying), or a post-crossfade press folds against the wrong (stopped) node and the rewind returns. A genuine continuous-playback press is the only case that folds.

Persistence and overrides

UserDefaults keys (mirrors EQ/volume normalization pattern):

• referenceTuningEnabled: Bool
• referenceTuningSourceHz: Double (default 440)
• referenceTuningTargetHz: Double (default 432)
• playbackSpeedRate: Float (default 1.0; UserDefaults.float(forKey:) == 0 means missing/default)

CLI flags (--tuning, --tuning-source, --tuning-offset-cents) provide session-only overrides — they do not write back to UserDefaults, matching --volume and --eq.

Casting

Casting paths (Sonos / Chromecast / DLNA) hand the remote renderer a stream URL with no local AVFoundation graph to insert into, so Reference Tuning and Playback Speed are intentionally unavailable while casting. Their menus grey out with a "Not available while casting" tooltip; engine.isAnyCastingActive is the gate. The persisted speed remains stored and resumes for local/HTTP playback after casting ends.

Spectrum Analyzer

Both pipelines feed spectrum data to the UI for visualization.

Processing Pipeline

1. Sample extraction - Get float samples from PCM buffer (mono-mix stereo)
2. Windowing - Apply Hann window to reduce spectral leakage
3. FFT - 2048-point DFT using Accelerate framework
4. Magnitude calculation - Convert complex output to magnitudes
5. Band aggregation - Accurate mode is pipeline-specific; adaptive/dynamic modes interpolate at band center frequencies
6. Frequency weighting - Apply compensation curve (adaptive/dynamic modes)
7. Normalization - Scale based on selected mode
8. Smoothing - Fast attack, slow decay for visual appeal

Normalization Modes

Mode	Gain Control	Preserves Balance	Best For
Accurate	Fixed dB mapping	Yes (true levels)	Technical analysis
Adaptive	Global adaptive	Yes (scaled together)	General listening
Dynamic	Per-region (bass/mid/treble)	No (independent)	Visual appeal

Shared Processing And Accurate-Mode Difference

Both local and streaming use a 2048-point FFT and 75 logarithmic output bands:

• FFT size: 2048 samples
• Bin width: ~21.5 Hz at 44.1kHz
• Latency: ~46ms

Accurate mode is not identical across pipelines:

• Local AudioEngine: BeSpec-style peak aggregation per band, calibrated by 2 / sqrt(fftSize), mapped from -20...0 dB.
• Streaming StreamingAudioPlayer: RMS power integration per band with bandwidth compensation, mapped from 0...40 dB.

Adaptive and dynamic modes use the same algorithm shape in both pipelines: center-bin interpolation, bandwidth scaling, frequency weighting, adaptive normalization, and fast-attack/slow-decay smoothing.

Volume-Independent Visualizations

Spectrum analyzer and ProjectM show audio levels independently of user volume:

Local: Tap on mixerNode before volume control (mainMixerNode.outputVolume)

Streaming: AudioStreaming's frameFiltering captures after volume, so processAudioBuffer() compensates by dividing samples by current volume (capped at 20x).

Waveform Window Pipeline

The waveform window shares the audio engine but intentionally does not share the same demand gate as FFT/spectrum analysis.

Local Files

• WaveformCacheService opens the active file with AVAudioFile
• Decodes PCM in chunks and stores max absolute amplitude into 4096 buckets
• Persists snapshots under ~/Library/Application Support/NullPlayer/WaveformCache/
• Cache key is based on canonical path + file size + modification date

Streams

• Live path:
  • AudioEngine and StreamingAudioPlayer emit .audioWaveform576DataUpdated
  • BaseWaveformView listens only for non-file audio tracks
  • StreamingWaveformAccumulator builds progressive seekable waveforms for timed streams, and rolling non-seekable waveforms for live/radio streams
• Prerender path (service-backed tracks with stable identity + known duration):
  • WaveformCacheService can prerender remote waveforms and persist them as seekable snapshots
  • Service cache key: WaveformCacheService.serviceCacheKey(serviceIdentity:duration:bitrate:sampleRate:)
  • Live placeholders or unknown-duration streams skip prerender and stay on live accumulation
  • Generation tries AVAssetReader first, then falls back to URL download + local decode
  • Once a seekable service prerender is ready, BaseWaveformView freezes on that snapshot and ignores subsequent live 576-sample chunks for the same track

Consumer Gating

Do not assume spectrum demand implies waveform demand.

• AudioEngine.spectrumConsumers controls FFT/spectrum work
• AudioEngine.waveformConsumers controls 576-sample waveform chunk generation
• AudioEngine.stereoConsumers controls separate L/R downsampled PCM (.audioStereoPCMDataUpdated)
• AudioEngine.magnitudesConsumers controls raw linear FFT magnitudes (.audioFFTMagnitudesUpdated)
• The waveform window registers a waveform consumer only while visible on a non-file audio track
• SpectrumAnalyzerView registers a waveform consumer only while qualityMode == .visClassicExact and the analyzer is actively rendering
• The Audio Analyzer window registers per-pane consumers (see audio-analysis-window skill); a closed window deregisters all of them

This split matters for CPU usage: hidden waveform windows and inactive vis_classic views should not keep paying the live waveform callback cost.

Stereo PCM and FFT-magnitudes paths (Audio Analyzer)

Two analysis-only paths exist alongside the mono PCM/spectrum/waveform feeds, each gated by its own consumer set so it costs nothing when unused:

• .audioStereoPCMDataUpdated — ["left": [Float](512), "right": [Float](512), "sampleRate": Double]. Separate downsampled L/R buffers (mono streams are duplicated into both). Gated by stereoNeeded.
• .audioFFTMagnitudesUpdated — ["magnitudes": [Float](fftSize/2 linear), "sampleRate": Double, "fftSize": Int]. Raw linear FFT magnitudes (not dB, not the 75-band normalized spectrum). The FFT runs when spectrumNeeded || magnitudesNeeded; this notification is posted right after the magnitudes are computed, gated by magnitudesNeeded, and the 75-band spectrum work is gated separately by spectrumNeeded. So a magnitudes consumer is independent — it does not require a spectrum consumer.

Both paths are wired in AudioEngine (local) and StreamingAudioPlayer (streaming) and emit the identical payload shape. A consumer listening to only one path silently misses the other engine — the stereoNeeded/magnitudesNeeded mirror flags bridge engine → streaming the same way spectrumNeeded does.

Consuming tap notifications safely — never observe on queue: .main ⚠️

.audioPCMDataUpdated, .audioStereoPCMDataUpdated, .audioFFTMagnitudesUpdated, and .audioWaveform576DataUpdated are posted directly from the realtime audio tap thread (see AudioEngine.processAudioBuffer; .audioSpectrumDataUpdated is the exception — it is dispatched to main at the source). NotificationCenter delivers a queue:-based observer synchronously: it enqueues an NSOperation on that queue and calls waitUntilFinished. So an observer registered with queue: .main blocks the audio tap thread on the main queue.

That deadlocks against tap teardown: AudioEngine.commitLoadedLocalTrack calls mixerNode.removeTap(onBus:) on the main thread, which waits on the AVFoundation RealtimeMessenger lock the tap thread holds while it is blocked posting to main. Rapid track loads (e.g. spamming Play) make teardown coincide with a tap callback and hang the app.

Correct pattern for any window/view consuming these tap notifications — register with queue: nil (runs synchronously on the posting thread, no OperationQueue wait) and hop to main yourself:

observer = NotificationCenter.default.addObserver(
    forName: .audioPCMDataUpdated, object: nil, queue: nil   // NOT .main
) { note in
    guard let pcm = note.userInfo?["pcm"] as? [Float] else { return }
    DispatchQueue.main.async { [weak self] in
        self?.ingest(pcm)   // UI / mutable state touched only on main
    }
}

This is poster-agnostic, so it covers both the AudioEngine and StreamingAudioPlayer posting paths. Reference implementations: BaseWaveformView, PeppyMeterLevelModel, ScopePaneView, OctavePaneView. Views needing lowest latency (e.g. ProjectMView, SpectrumAnalyzerView) instead do their work directly on the posting thread with their own locking — also queue: nil, just no main hop. Either way, never queue: .main for a tap-posted notification.

BPM Detection

Real-time BPM detection using aubio library (libaubio.5.dylib):