Learning to Separate from Mislabeled Stems with Sum-Preserving Noise Models

Music separation models learn to split a song into stems such as vocals, drums, bass and other. They are trained on multitrack recordings in which each stem is assumed to contain only its own instrument. Real multitrack collections break that assumption in two common ways. Stems get mislabeled or swapped, and microphones pick up bleed, for example drums audible on the vocal track. Both corruptions have a property that has gone unused: the corrupted stems still add up exactly to the song. A swap only exchanges two stems. Bleed only moves drum energy from the drum stem into the vocal stem. The corruption changes how the mixture is divided among stems and leaves the mixture itself untouched.

This matters because the most widely used structural constraint in separation, mixture consistency (the rule that the estimated sources must sum back to the input), is blind to exactly this kind of error. Adding a signal to one source and subtracting it from another leaves the sum unchanged. So a mixture-consistent separator gets its entire learning signal from how energy is allocated among stems, and that allocation is precisely what the corruption damages. We propose to turn this weakness into a noise model. For each training track, we fit a small per-track "remixing matrix" that explains how the observed stems were redistributed from the true sources. The matrix is constrained so that it, too, preserves the sum. Because one separator is shared across all tracks, it learns what "vocals" means from the tracks that are mostly clean, while the per-track matrices absorb the swaps and bleed. No clean reference set is needed, and no data is thrown away.

Core Claim

Stem label noise in separation is sum-preserving noise, and it should be modeled as a latent, per-track, sum-preserving linear remix of the true sources. This is the signal-space analogue of the noise transition matrices used for noisy classification labels. The column-sum constraint is what makes the model faithful to the corruption and keeps it from swallowing arbitrary errors. Concretely, let $X_n(t,f)$ be the short-time spectrum of the mixture for track $n$. Let $\hat S_n(t,f) \in \mathbb{C}^K$ be the separator's $K$ source estimates, which sum to $X_n$ by construction. Let $\tilde Y_n(t,f)$ be the observed, possibly corrupted, stems, and let $A_n(f)$ be a $K \times K$ complex matrix for each frequency, held fixed over time within the track. Training minimizes

$$ \mathcal{L} = \sum_{n,t,f} \big\| \tilde Y_n(t,f) - A_n(f)\,\hat S_n(t,f) \big\|_1 + \lambda \sum_{n,f} \big\| A_n(f) - I \big\|_1 \quad \text{s.t.} \quad \mathbf{1}^\top A_n(f) = \mathbf{1}^\top , $$

where $I$ is the identity matrix, $\mathbf{1}$ is the all-ones vector, and $\lambda$ sets how strongly corruption is assumed to be rare and sparse. Because $\mathbf{1}^\top A_n = \mathbf{1}^\top$ and $\mathbf{1}^\top \hat S_n = X_n$, the predicted noisy stems $A_n \hat S_n$ also sum to $X_n$, just as the observed ones do. The noise model can therefore only move energy between stems, which is exactly what swaps and bleed do.

A permutation matrix is a valid $A_n$, which covers swaps. Off-diagonal frequency-dependent gains cover bleed, since bleed is approximately a fixed acoustic filter applied to the leaking instrument. At test time $A_n$ is discarded and only $\hat S$ is used.

Why This Gap Is Real

The closest work is Towards Blind Data Cleaning: A Case Study in Music Source Separation. It builds Mixed23, a set of 100 clean songs plus 50 with synthetic bleed and 50 with swapped stems. It scores tracks offline, either by unlearning a small clean reference set or by Fréchet Audio Distance to that set, and discards the worst quarter. This raised Demucs from 4.85 to 4.91 dB SDR on MDXDB21, against a 4.94 dB clean ceiling. That paper motivates our work: it shows the corruption matters and that detection is feasible. We differ on three counts. We correct corrupted tracks during training instead of discarding them. We require no clean reference set. And we exploit the sum-preserving structure of the corruption, which the paper's generic outlier scores ignore.

The label-noise literature supplies the mechanism, and it also warns us. Random Feature Amplification shows that gradient descent amplifies features shared across examples before it fits idiosyncratic ones. This is why a shared separator should learn "vocals" from the majority of tracks before per-track matrices are fitted. The constraint comes from Deep Neural Networks Learn Meta-Structures from Noisy Labels in Semantic Segmentation. Segmentation networks there shrug off random pixel flips but degrade sharply under systematic distortions such as dilation or erosion. Bleed is the audio version of a systematic distortion: it is correlated, rhythmic and synchronized with the song. We therefore should not expect the network's built-in robustness to handle it, which is why we model it explicitly.

On the separation side, three papers pair mixture consistency with weak supervision and assume that supervision is correct. Finding Strength in Weakness pairs a mixture reconstruction loss with a frozen event classifier. Weakly-supervised Audio Separation via Bi-modal Semantic Similarity pairs it with a text-audio embedding loss. Joint Optimization of Masks and Deep Recurrent Neural Networks builds consistency into masks that sum to one. All three treat mixture consistency as a guard against hallucinated content. None of them asks what happens when the allocation signal itself is corrupted, and that question is the one we address.

Research Program

We build on Demucs, the separator used by the blind-cleaning study, so that the numbers can be compared directly. Consistency is enforced by projecting the outputs so they sum to the mixture. The per-track matrices are parameterized per frequency band, using about 32 bands rather than every bin, to limit their capacity. The column-sum constraint is enforced exactly by writing the last row as one minus the sum of the others.

Training has three phases. First, a warm-up phase trains the separator alone on the raw stems for a short, fixed fraction of training. This exploits the early-learning effect. Second, the matrices are initialized to the identity and optimized jointly with the separator. Third, the fitted matrices are used for analysis: the off-diagonal mass of $A_n$ serves as a per-track corruption score. That score can be compared directly with the unlearning and distance scores of the blind-cleaning paper.

Beyond benchmark numbers, we will study identifiability. The central risk is that the matrices absorb genuine separator errors rather than label corruption, the same trade-off noise transition methods face in classification. We will measure this on synthetic corruption where the true remix is known. We will vary $\lambda$, the fraction of corrupted tracks, and the bleed gain, and report how well the fitted $A_n$ matches the true one. We will also locate the point at which too many tracks are corrupted for the shared separator to learn from a clean majority.

Baselines

The naive reference point is Demucs trained on the dirty data with the standard loss and consistency projection, with no noise handling. On Mixed23 evaluated on MDXDB21 the blind-cleaning paper reports 4.85 dB SDR for this setting. We will reproduce it under our own seeds before comparing anything.

The strongest existing method is offline blind data cleaning from Towards Blind Data Cleaning, with either unlearning attribution or Fréchet Audio Distance filtering at the reported best retention of about 75 percent. The paper reports 4.91 dB SDR on MDXDB21 for both, against a 4.94 dB clean upper bound. It is the strongest competitor because it targets exactly this corruption on exactly this benchmark. It also has an advantage we do not: a clean reference set.

We will run it with the same Demucs configuration, the same number of training steps on the retained data, and the same number of hyperparameter trials. Its tuned value is the retention ratio; ours is $\lambda$ and the warm-up length. To avoid giving our method hidden clean supervision, $\lambda$ and warm-up length are chosen on a synthetically corrupted MUSDB18 split and then frozen before Mixed23 is touched.

Beating this baseline means exceeding 4.91 dB on MDXDB21 across at least three seeds with non-overlapping confidence intervals. The gap on Mixed23 is only 0.09 dB, so this benchmark alone is weak evidence. The decisive comparisons are on harsher synthetic corruption, described below.

Two further baselines address simpler robust-training ideas. The first is online small-loss truncation: in each batch, drop the gradient from the highest-loss fraction of tracks, with that fraction tuned under the same trial budget. The second is per-track permutation-invariant training, which takes the minimum loss over stem permutations for each track and is equivalent to restricting $A_n$ to permutation matrices.

Finally, two ablations isolate the key ingredient. In the first, $A_n$ is unconstrained, so it no longer has to preserve the sum. If the column-sum constraint is what makes the noise model faithful, this variant should absorb real separator errors, recover $A_n$ worse, and lose SDR. In the second, $A_n$ is a single frequency-independent gain matrix. This tests whether modeling bleed as a filter is needed.

Why not just…?

Why not just clean the data offline, as the blind-cleaning paper does? Offline cleaning is a genuine competitor on Mixed23, where it already closes about two thirds of a small gap. However, it needs a trusted clean reference set, and in their experiments it degrades once more than half the data is pruned. That is the regime where corruption is common, which is where our method should help most. It also discards the vocal content of bled tracks, which is mostly correct. The corruption-rate sweep below will show whether its advantage vanishes as corruption rises. If it does not, we will report that plainly.

Why not just drop high-loss tracks online, or use a robust loss? For swaps this is a real threat. A swapped track produces a very large loss and is easy to truncate, so we expect small-loss truncation to roughly match our method on swap-only corruption. The difference should appear with bleed. Bleed at realistic gains yields only a modestly elevated loss on a track whose other stems are fine. The meta-structure results of Luo et al. suggest that such systematic noise is learned rather than rejected. We will report swap-only and bleed-only conditions separately so this is settled rather than averaged away.

Why not just use permutation-invariant training on the stems? Taking the minimum loss over permutations fixes swaps exactly and needs no new parameters. It is included as a baseline and is literally our method restricted to permutation matrices. It cannot represent bleed, because bleed is partial, frequency-dependent redistribution. It may also hurt clean training by letting similar stems, such as bass and other, trade identities. If it matches the full method on bleed conditions, the continuous matrix is unnecessary and we will say so.

Why not just trust the network's built-in robustness to label noise? Networks do tolerate random label noise remarkably well. However, Luo et al. show that they absorb systematic distortions. The weak-supervision separation results of Pishdadian et al. show a related failure on co-occurring bass and drums. The do-nothing baseline measures this directly across bleed gains.

Proposed Experiments

The primary testbed is MUSDB18 with controlled corruption. The corrupted-track fraction is swept from 0 to 60 percent. The corruption type is swap-only, bleed-only, or mixed. Bleed is simulated as a filtered, attenuated copy of the drums (or bass) added to the vocal (or other) stem and subtracted from the source stem, which keeps the mixture unchanged. The bleed gain is swept from minus 30 to minus 10 dB. All models are evaluated by SDR on the clean MUSDB18 test set. The same protocol is then applied unchanged to Mixed23 with MDXDB21 evaluation, so our results sit next to the published 4.85, 4.91 and 4.94 dB figures.

Four checks are decisive. First, on clean data the fitted matrices should stay near the identity and SDR should not drop relative to plain training; a method that hurts clean training is not viable. Second, on bleed-only corruption the full method should beat small-loss truncation, and the gap should widen with bleed gain. Third, the unconstrained-matrix ablation should recover the true remix worse and lose SDR, which would confirm that sum preservation is doing the work. Fourth, off-diagonal mass in $A_n$, used as a corruption detector, should reach an area under the ROC curve at least comparable to the reference-based scores of the blind-cleaning paper, without seeing any clean data. For this detector comparison, the blind-cleaning paper reports end-task SDR rather than detection accuracy. We will therefore rerun its scoring ourselves, and no reported detection result was found on Lacuna for it.

If the second and third checks fail, the conclusion is that per-track permutation handling or loss truncation suffices for stem noise in separation. That would still be a useful negative result, because it would show that the sum-preserving structure buys nothing in practice.