When two instruments play at once, some of their energy lands in the same time-frequency cells of the spectrogram, the picture of which frequencies are loud at each moment. A separation model has to decide who gets that shared energy. We argue that current models settle this in favor of whichever instrument was more common in training, and that this is the audio version of popularity bias in recommender systems, where frequently seen items win rankings because the model has seen them more, not because they fit better. The practical result is familiar. Rare instruments such as bassoon or viola are quietly swallowed by piano or strings, and transcription systems "hallucinate" common instruments when rare ones are actually playing.
We propose to measure this effect directly and to correct it with one targeted training term. The model is a non-negative matrix factorization (NMF) separator. NMF writes a spectrogram as a sum of non-negative spectral templates times non-negative activations, so every unit of energy is assigned to exactly one instrument's templates and no instrument can cancel another. That additivity lets us build an attribution matrix over only the cells where instruments collide. The matrix records how much each instrument's predicted output tracks each true isolated track (stem). The corrective term is a redundancy-reduction loss in the style of Barlow Twins, with one key change: it pulls the attribution matrix toward the correlation structure of the true stems rather than toward the identity.
Core Claim
In supervised non-negative separation of polyphonic music, a common instrument's predicted output will correlate with a rare instrument's true stem more than the two true stems correlate with each other. We call that surplus leakage excess. We expect it to grow with the log ratio of the two instruments' training frequencies, and to be concentrated in the small fraction of cells where stems overlap. A loss that removes the surplus in those cells should shrink the head-tail performance gap more than energy-normalized or frequency-reweighted per-stem losses do. It should do so without the global accuracy loss that aggressive debiasing tends to cause.
Let $\Omega$ be the set of time-frequency cells where at least two true stems each carry more than a fraction $\rho$ of the mixture energy. Let $\hat S_i$ be instrument $i$'s predicted non-negative contribution $W_i H_i$, and let $S_j$ be the true magnitude stem of instrument $j$. Write $C_{ij}$ for the Pearson correlation over $\Omega$ between $\hat S_i$ and $S_j$, and $R_{ij}$ for the correlation over $\Omega$ between $S_i$ and $S_j$. The loss is
$$\mathcal{L}_{\text{CZA}} = \sum_i (1 - C_{ii})^2 + \lambda \sum_{i \neq j} (C_{ij} - R_{ij})^2 + \mu \sum_i \max(0, \gamma - \sigma(\hat S_i))^2,$$
where the last term is a variance floor on each instrument's predicted output, borrowed from VICReg, with threshold $\gamma$ and standard deviation $\sigma$.
Where This Sits
The mechanism has three sources.
Barlow Twins drives the cross-correlation between two augmented views of an embedding toward the identity. VICReg adds per-feature variance and covariance penalties. DeCov penalizes hidden-unit covariance in supervised networks and shows it helps models ignore injected spurious correlations. We reuse their cross-correlation machinery, but apply it between predictions and ground truth rather than between views.
We also correct their target. The research file argues that non-negativity rules out decorrelation. That is only half right. Non-negative variables can have zero covariance. The real obstacle is that instruments genuinely co-occur, so an identity target would punish correct behavior. Referencing the target to the true stem correlations $R$ removes exactly the correlation the model invented and keeps the correlation that is really in the music.
The popularity lens comes from recommendation. Adap-τ shows that frequent items' embedding norms grow with their gradient volume and dominate dot-product scores, and fixes this with normalization plus adaptive temperature. MACR separates a direct popularity effect from true matching and subtracts it at inference. Both work on discrete interaction matrices. Neither faces a setting where the "items" physically share the same measurement cell. Our dictionary analogue of norm growth is that common instruments acquire more and broader templates that also cover rare instruments' spectra. We test this directly with an equal atom budget per instrument and normalized template norms.
The evidence for the gap comes from several papers. A Hierarchical Deep Learning Approach for Minority Instrument Detection finds fine-grained F1 stuck near 0.45 on MedleyDB even with focal loss or hierarchy. The 2025 AMT Challenge reports that the top system falls from 0.7193 F-measure on solo excerpts to 0.4585 on multi-instrument ones. MT3 is known for instrument leakage. Two papers set the constraint that the correction must work where overlap is dense and not just on average: DGMO and Fast accuracy estimation of deep learning based multi-class musical source separation argue that magnitude masks hit a ceiling in shared cells. Jointist shows that symbolic guidance helps separation under collision, but it treats Slakh's imbalanced classes uniformly. Listen to Interpret uses Sparse-NMF only as a post-hoc explainer. We train the NMF structure end to end as the separator itself.
Research Program
The separator is a convolutional encoder that reads the log-magnitude mixture and outputs non-negative activations $H_i$ for each Slakh2100 instrument class. Each class has its own learned non-negative dictionary $W_i$, with columns normalized and a fixed number of atoms per class. Predicted stems are $W_i H_i$, rescaled by a soft ratio so they sum exactly to the mixture. The base training loss is per-stem magnitude reconstruction.
The study has three stages. The first is diagnostic. We train the base model on Slakh2100 (Cutting Music Source Separation Some Slakh), compute the leakage-excess matrix $C - R$ on held-out collision cells, and regress each entry on the log training-frequency ratio of the pair. The second stage adds $\mathcal{L}_{\text{CZA}}$.
Small batches make rare-pair correlations noisy, a limitation of decorrelation losses noted in the file. We address this in two ways. We estimate $C$ and $R$ with running moment averages across batches. We also sample training excerpts so that every instrument pair seen in the training data appears in collision cells at a minimum rate. The third stage tests transfer. The trained separator feeds a per-instrument activity detector on MedleyDB, and its separated stems feed a transcription model on the 2025 AMT Challenge set.
Baselines
The naive reference is the same NMF separator trained with plain per-stem reconstruction. It gives the uncorrected leakage matrix and head-tail gap that everything else is measured against.
The strongest simple competitor is the same separator trained with a per-stem loss that is both energy-normalized (scale-invariant signal-to-distortion ratio per stem) and weighted by inverse instrument frequency. It uses focal weighting on the activity head, following the loss comparison in the hierarchical minority-instrument paper. This baseline attacks both energy dominance and frequency dominance with no new machinery, so it is the real test. It gets the same architecture, data, steps, and a hyperparameter search of equal size to ours. Our method beats it only if tail-decile separation quality in collision cells and tail-instrument F1 both improve, and head-decile quality drops by less than 0.5 dB.
The strongest existing system for separation under collision is Jointist. Its reported result is relative, a 5 dB SDR gain over unguided separation on Slakh2100. No absolute per-instrument Slakh SDR was reported in the file, so we will retrain it on our splits with its released recipe and its staged pretraining. Adding $\mathcal{L}_{\text{CZA}}$ to Jointist's separation module is also a test of whether the term helps beyond our NMF model.
For detection, the reference is the hierarchical system's reported 0.45 fine-grained F1 on MedleyDB, with 0.43 for focal loss and 0.41 for class weighting. For transcription, the references are MIROS at 0.5998 overall F1 and MT3 at 0.3932 on the 2025 AMT Challenge set.
Three ablations isolate the key ingredient. The first computes the loss over all cells instead of only $\Omega$. The second uses the identity target instead of $R$, which is plain Barlow Twins; we expect it to suppress genuine co-occurrence. The third keeps only the Adap-τ-style equal atom budget and norm control and drops the correlation term. A fourth ablation replaces the NMF head with an unconstrained ratio-mask head under the same loss, to check whether non-negative factorization is actually needed.
Why not just…?
Why not just reweight rare instruments or use focal loss? Reweighting changes how much each instrument's error counts, but a stolen unit of energy in a collision cell makes up a tiny share of any stem's total error. The file's evidence is also discouraging: focal loss moved MedleyDB fine-grained F1 only from 0.41 to 0.43. MACR reports that propensity reweighting is high-variance and hard to tune. Still, this is the baseline most likely to match us, which is why it gets the full tuning budget. If it closes the leakage-excess slope as well as our loss does, the claim fails.
Why not just normalize the dictionaries, as NMF practitioners already do? Column normalization is standard, and it is our third ablation. Adap-τ's analysis suggests norm control addresses magnitude but not span. A common instrument can still hold more templates that fit rare spectra. We expect norm control alone to reduce but not remove the frequency-dependent slope. If it removes the slope, the correlation term is unnecessary and we will report that.
Why not just use a large end-to-end transcriber like MIROS or MT3? These systems are strong overall, but the AMT Challenge's drop from solo to multi-instrument performance shows that scale has not solved collision. They also expose no per-instrument energy accounting in which leakage could be measured or penalized. We test complementarity rather than replacement by feeding our separated stems to the transcriber.
Proposed Experiments
The decisive check is the slope of leakage excess against log frequency ratio on Slakh2100 test collision cells. The diagnostic succeeds if the naive model shows a significantly positive slope, and the method succeeds if that slope becomes statistically indistinguishable from zero while the diagonal $C_{ii}$ does not fall.
Secondary outcomes are per-instrument SI-SDR, reported by training-frequency decile and restricted to $\Omega$, alongside global SDR. This addresses the known risk that debiasing hurts overall accuracy. For transfer, we report MedleyDB fine-grained instrument F1 against the 0.45 reference, URMP per-instrument separation, and 2025 AMT Challenge multi-instrument F-measure against 0.4585.
A counterfactual probe follows MACR's logic. We remove one rare stem from a test mixture and measure how much energy the model still assigns to common instruments in that stem's former cells. This isolates popularity-driven attribution from genuine acoustic ambiguity. The ratio-mask ablation settles whether the non-negative factorization structure matters or whether the attribution loss alone does the work.
References
- Barlow Twins: Self-Supervised Learning via Redundancy Reduction
- VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning
- Reducing Overfitting in Deep Networks by Decorrelating Representations Michael Cogswell
- Adap-τ : Adaptively Modulating Embedding Magnitude for Recommendation
- Model-Agnostic Counterfactual Reasoning for Eliminating Popularity Bias in Recommender System
- A Hierarchical Deep Learning Approach for Minority Instrument Detection
- Advancing Multi-Instrument Music Transcription: Results from the 2025 AMT Challenge
- MT3: Multi-Task Multitrack Music Transcription
- DGMO: Training-Free Audio Source Separation through Diffusion-Guided Mask Optimization
- Fast accuracy estimation of deep learning based multi-class musical source separation
- Jointist: Simultaneous Improvement of Multi-instrument Transcription and Music Source Separation via Joint Training
- Listen to Interpret: Post-hoc Interpretability for Audio Networks with NMF
- Cutting Music Source Separation Some Slakh: A Dataset to Study the Impact of Training Data Quality and Quantity