Real-time music source separation pulls one instrument or voice out of a live mix within about 25 milliseconds. It powers stage monitoring, hearing assistance and live remixing, but its outputs carry "bleed": faint remnants of the other instruments left inside the extracted stem. The bleed does not stay put. Models trained on separated stems learn it as part of the target. Accompaniment generators reproduce it, and deepfake detectors get confused by it. We argue that at low latency, most of this bleed comes from the separator's limited view along the frequency axis, not from its limited view along time. A bass note's overtones land many frequency bins above its fundamental. A network that only mixes information between neighboring frequency bins cannot tell that energy at a high bin belongs to the bass rather than the voice.
Within a single short audio frame, the network can see every frequency at once without waiting for future audio. So frequency-wide context costs essentially no extra latency, unlike context along time. The proposal is a controlled study and a resulting model. We hold the latency budget and the causal time-processing module fixed and vary only how far each frequency bin can "see" across the spectrum. We measure bleed with a metric that is not fooled by imperfect references. We then test whether lower bleed actually reduces what downstream models inherit.
Core Claim
At a fixed algorithmic latency of about 23–25 ms, giving every frequency bin access to the full spectrum of its own frame substantially reduces interference energy in the separated stem. It should do so at small cost in parameters and CPU runtime. The reduction should be concentrated in partials whose generating fundamental lies far from them in frequency.
If this holds, two things follow. First, it explains part of the gap between RT-STT's 5.17 dB and Band-SCNet's 7.79 dB on MUSDB18-HQ. Second, it directly contests RT-STT's design argument that frequency modeling is wasted under short analysis windows.
The claim makes a measurable prediction that a temporal-context explanation does not make. Bleed in local-frequency models should grow with the bin distance between an interfering fundamental and the leaking overtone. Full-spectrum models should flatten that curve.
Where This Sits in Prior Work
The motivating evidence comes from the artifact literature. FastSAG trained a fast diffusion accompaniment generator on Demucs-separated vocals and reports that separation residue leaked into its outputs. Text-to-Song observed memorized inter-track bleeding. SingFake found that pre-separating vocals did not reliably help deepfake detection, which it attributes to separation artifacts. How Does Instrumental Music Help SingFake Detection? localized detection cues to 0–0.8 kHz. That band is exactly where harmonics of bass and accompaniment overlap the vocal fundamental. All of these diagnose the problem downstream; none changes the separator.
The measurement problem is set by Music Demixing Challenge 2021. It notes that MUSDB18 reference stems themselves contain studio bleed, so the standard quality score (signal-to-distortion ratio, SDR) can penalize a separator for removing it. This constrains our design: we cannot rely on SDR against MUSDB18 references to measure bleed.
The real-time separators define the design space we test. RT-STT drops frequency modeling in favor of a single temporal LSTM and reaches 23 ms latency with 1.01 ms GPU inference, but only 5.17 dB SDR. Band-SCNet reaches 7.79 dB, but it needs a 92 ms window. Its "cross-band" blocks are convolutions over neighboring bands, so its frequency view is still local.
The mechanism comes from speech enhancement and vision. RT-SEMamba runs a state space model across all frequency bins in both directions within a frame, and causally across time. It does this at 25 ms latency, but only for speech denoising on Voicebank-DEMAND. A state space model (SSM) is a recurrent layer whose cost grows linearly with sequence length. CCNet gets global image context cheaply by attending along one axis at a time. Butterworth as Attention shows learnable gating in the Fourier domain gives a global receptive field at $O(N \log N)$ cost.
Our difference from RT-SEMamba is not the block, which we reuse. It is three things it does not do: isolate frequency-axis context as the causal variable for bleed in polyphonic music, measure bleed with clean references, and test downstream propagation.
Research Program
We build one causal separator skeleton at a 1024-point window at 44.1 kHz, which gives roughly 23 ms of algorithmic latency to match RT-STT. It has a causal convolutional encoder and decoder and a fixed causal temporal module. That module is a unidirectional SSM across frames, run with its LSTM counterpart as a check.
Between the temporal layers sits a swappable frequency module. Its frequency reach ranges from none at all, to local, to full-spectrum. The six variants are:
- None: no frequency module, as in RT-STT.
- Local: a 3-bin convolution along frequency.
- Band-local: Band-SCNet-style neighboring-band convolution with sparse downsampling.
- Dilated: a stack of dilated frequency convolutions whose reach spans the whole spectrum.
- Full-spectrum SSM: a bidirectional within-frame frequency SSM in the RT-SEMamba style.
- Fourier gating: a learnable gate applied in the Fourier domain along frequency, following the Butterworth-gating idea.
All variants are matched in parameter count, within 10% of 1M, and in FLOPs per frame. All are trained with the same data, schedule and tuning budget. Any difference is therefore attributable to frequency reach and to the kind of operator that provides it.
Bleed is measured on mixtures whose vocal references are genuinely clean. We build them by mixing a cappella recordings from ACappellaSet and GTSinger with MUSDB18-HQ instrumental stems from different songs. Following the standard BSS-Eval decomposition, the estimated vocal is split into a target part, an interference part (projected onto the known accompaniment), and an artifact part. Our main metric is the interference-to-target energy ratio per band:
$$ L_b = 10 \log_{10} \frac{\| P_b\, e_{\text{interf}} \|^2}{\| P_b\, s_{\text{target}} \|^2} $$
Here $e_{\text{interf}}$ is the interference component of the estimate, $s_{\text{target}}$ is the target component, and $P_b$ restricts a signal to frequency band $b$. We report $L_b$ for 0–0.8 kHz separately, since that is the band the deepfake probing work flags.
A diagnostic probe set uses synthetic harmonic interferers: bass-like tones with a controlled fundamental under a sung vowel. It lets us plot interference energy against the bin distance between each interfering overtone and its fundamental. This curve is the signature that separates our explanation from a temporal-context one.
Baselines
The naive reference is the unprocessed mixture taken as the vocal estimate, together with the "None" variant of our skeleton. No reported result was found on Lacuna for the mixture-as-estimate on MUSDB18-HQ. We compute it ourselves to anchor $L_b$ and SDR.
The strongest existing method at our latency is RT-STT. It reports 5.17 dB average SDR on MUSDB18-HQ at 23 ms with 383K parameters (Towards Practical Real-Time Low-Latency Music Source Separation). It is the right competitor because it occupies the same latency budget and explicitly argues against frequency modeling. We retrain it on our data with its published configuration and the same tuning budget. We also run a width-scaled version at our 1M parameter budget, so parameter count cannot explain the outcome. Beating it means three things: higher MUSDB18-HQ SDR, at least 1 dB lower 0–0.8 kHz $L_b$ on clean-reference mixtures, and CPU real-time factor below 1.
The strongest method at any latency is Band-SCNet. It reports 7.79 dB SDR, 92 ms latency and RTF 0.478 on an Intel i7 CPU (Band-SCNet). We run it both at its native 4096-point window and retrained at our 1024-point window. That second run tells us whether its advantage survives at 23 ms. A full-spectrum variant at 23 ms that matches 92 ms Band-SCNet on $L_b$ would be a strong result. Approaching its SDR is a stretch goal, not a claim.
HS-TasNet's 4.65 dB with 42M parameters, as reported in the same two papers, serves as a dilated-convolution reference point.
The ablation that isolates the key ingredient is the frequency-module sweep itself. The comparison between the dilated-convolution arm and the full-spectrum SSM arm separates "full reach" from "this particular operator."
Why not just…?
Why not just use dilated frequency convolutions, which also reach the whole spectrum cheaply? This is a real threat, and the reason the dilated arm exists. If it matches the full-spectrum SSM on $L_b$ and the harmonic-distance curve, our conclusion becomes "full frequency reach matters," not "SSMs matter." We would then recommend the dilated design for its better behavior on CPUs that lack fused scan kernels, a limitation RT-SEMamba itself notes. We regard this outcome as a successful answer, not a failure.
Why not just lengthen the window or add look-ahead, as Band-SCNet does? That buys frequency resolution with latency. Band-SCNet's 92 ms already exceeds the roughly 25 ms interactive threshold. Retraining Band-SCNet at 1024 points shows how much of its gain depends on the long window. If full-spectrum context at 23 ms recovers most of that gain, the latency trade is unnecessary.
Why not just distill a large offline separator into a small causal student, as RT-SEMamba does for speech? Distillation transfers the teacher's outputs. But an offline teacher trained on MUSDB18 references inherits their bleed. A student with only local frequency reach may also be unable to represent the teacher's cross-band decisions. We test this by distilling the same teacher into the "Local" and "Full-spectrum SSM" students. If both reach equal $L_b$, distillation alone suffices and our structural claim is wrong.
Why not just clean separated audio after the fact with classical filters? fSER is cheap but uses fixed contrast curves tuned by hand for speech in stationary noise. Bleed is correlated with the accompaniment and changes over time. We include spectral subtraction using the separator's own accompaniment estimate as a post-filter on the "None" variant, so this is checked rather than assumed.
Proposed Experiments and Decisive Checks
The first decisive check is the harmonic-distance curve on the synthetic probe set. The local-frequency arms should show interference rising once the interferer's fundamental lies beyond their frequency reach; the full-reach arms should not. If all arms show the same curve, the frequency-context explanation is false, and we redirect toward temporal context.
The second check is clean-reference $L_b$ on the ACappellaSet and GTSinger mixtures, alongside standard MUSDB18-HQ SDR. We would also report it on SingFake vocals if their references allow, but they do not, so we use only the clean sets. We expect $L_b$ and SDR rankings to disagree in places, which would be a finding about the evaluation itself.
The third check is downstream propagation. Following SingFake's protocol, we train the Wav2vec2+AASIST detector on vocals separated by each arm and by an offline Demucs reference. The SingFake paper reports 8.23% equal error rate (EER) on T02 mixtures for this detector. We report EER on T02 and T03. We also run the ASR-probing protocol from How Does Instrumental Music Help SingFake Detection?, which measures how much the encoder's speech-recognition ability degrades. The prediction is that detection gaps against raw mixtures, and ASR degradation, shrink as $L_b$ falls.
A weaker link between $L_b$ and downstream EER is plausible, because detectors also exploit other cues. That result would still be informative: it would say separator bleed is not the main channel of artifact propagation.
All timing is reported as RTF on a desktop CPU and on an ARM mobile CPU, in FP16, with INT8 quantization-aware training as a secondary condition. This follows RT-STT's finding that post-training INT8 quantization is fragile in small models.
References
- FastSAG: Towards Fast Non-Autoregressive Singing Accompaniment Generation
- Text-to-Song: Towards Controllable Music Generation Incorporating Vocal and Accompaniment
- SingFake: Singing Voice Deepfake Detection
- How Does Instrumental Music Help SingFake Detection?
- Music Demixing Challenge 2021
- Towards Practical Real-Time Low-Latency Music Source Separation
- Band-SCNet: A Causal, Lightweight Model for High-Performance Real-Time Music Source Separation
- RT-SEMamba: Real-Time Speech Enhancement Mamba via Progressive Knowledge Distillation
- CCNet: Criss-Cross Attention for Semantic Segmentation
- Butterworth as Attention: Anisotropic Spectral Gating for Pansharpening
- ACappellaSet: A Multilingual A Cappella Dataset for Source Separation and AI-assisted Rehearsal Tools
- A fast algorithm for improved intelligibility of speech-in-noise based on frequency and time domain energy reallocation