Music mood taggers, the models that label a track "happy," "dark," or "epic," are known to cheat. They pick up how a track was recorded and mixed, such as its loudness, how heavily it was compressed, and how bright the equalization is, because in training data those recording traits travel together with the mood labels. The usual fix is to make the model ignore production through augmentation or invariance. That fix assumes production is pure noise, and for mood it is not. A heavily compressed, loud master plausibly does sound more energetic to real listeners. The question this proposal asks is therefore not "is the model sensitive to production?" but "is the model sensitive to production by as much as people are, and no more?"
We propose to answer that question with propensity score matching, a technique from observational statistics. It compares "treated" and "untreated" units only after pairing them so that their other characteristics are balanced. Here the treatment is a production trait, for example a high versus low loudness range. Units are tracks, matched on their musical content. Within matched pairs we estimate two quantities: how much the human tags change with production, and how much the model's scores change. The gap between them is the model's excess production reliance. We then train taggers so that their response to controlled production edits matches the human-estimated response, rather than forcing that response to zero. A small randomized listening test, in which we re-master the same excerpts and collect ratings, checks whether the observational estimate is trustworthy.
Core Claim
The claim has two parts. First, a production-matched contrast, the matched human effect minus the matched model effect, is a better diagnostic of shortcut learning in subjective tagging than either artist-split accuracy or post-hoc feature attribution. It separates production effects that listeners actually perceive from those that are artifacts of how the dataset was assembled. Second, a training objective that pushes a model's production sensitivity toward that human target will reduce excess reliance and improve transfer to differently produced music, without the accuracy loss that full invariance should cause on tags where production genuinely matters.
The subjectivity direction is the reason for the first claim. Mood tags carry no single ground truth, as The MediaEval 2013 Brave New Task: Emotion in Music argued when it showed that averaging listeners erases real disagreement. A second confound follows: in collections tagged at upload time, the person producing the track and the person labeling it are often the same artist. The artist's mixing habits and tagging vocabulary are then correlated through a hidden common cause. Artist-controlled splits, as in MediaEval 2021: Emotion and Theme Recognition in Music Using Jamendo and FMA, keep one artist's tracks on one side of the split. They do not stop a model from learning genre-wide production conventions shared across many artists.
Relation to Prior Work
Three papers motivate the gap. Challenges and Perspectives in Interpretable Music Auto-Tagging Using Perceptual Features built an XGBoost tagger on harmony, perceptual, and signal features and used SHAP to show it leans on production-like cues such as spectral flux and loudness. It diagnoses through attribution, which describes the model's behavior but cannot say whether listeners share that dependence. Music Artist Classification with Convolutional Recurrent Neural Networks showed that random splits inflate accuracy through album leakage. A Data-Driven Analysis of Robust Automatic Piano Transcription showed high-capacity models overfitting to room and microphone. Both concern objective targets, where any production sensitivity is a bug. That premise fails for mood.
Three papers supply method. Improving Generalizability of Fake News Detection Methods using Propensity Score Matching treats each word as a treatment, matches documents on context, and keeps words with matched effects. This raised cross-domain AUROC from 0.56 to 0.68 from PolitiFact to GossipCop. We borrow the move of treating one feature as an intervention and matching on the rest. Our setting differs in two ways: it uses continuous audio descriptors rather than tokens, and it compares a model's effect against a human effect instead of selecting features. Reasoning about Political Bias in Content Moderation used matching to test a conditional-independence criterion. A raw 4.20% versus 2.35% disparity collapsed to about 2.5% once comments were matched on justifiable features. Our excess-reliance metric is the audio analogue: the "justifiable" production effect is the human one. A Data-driven Approach to Mid-level Perceptual Musical Feature Modeling supplies the musical-content covariates. Its seven perceptual dimensions, including melodiousness, dissonance, and modality, predict valence at r = 0.88. That makes them a compact, meaningful thing to match on. Towards Explainable Music Emotion Recognition shows these dimensions can be predicted from audio inside an emotion model at little cost.
Two papers constrain the design. Improving Weakly Supervised Sound Event Detection with Causal Intervention applies backdoor adjustment over a pooled acoustic context and gains about one mAP point on DCASE2017. It targets context invariance for objective events, the stance we argue is wrong for mood. Addressing the confounds of accompaniments in singer identification shows that synthetic remixing helps (F1 from 0.67 to 0.75 on artist20) but brings in separation artifacts. We therefore restrict our interventions to mastering-chain edits, namely gain, compression, equalization tilt, and added reverb. These change production without separating sources.
Research Program
The first step defines production descriptors. A trait counts as "production" only if a mastering chain can change it without changing the notes, so that the observational treatment and the later controlled intervention refer to the same thing. The descriptors are integrated loudness, loudness range (a proxy for compression), crest factor, spectral tilt, and an estimate of reverberation. Spectral centroid is excluded from the treatment set because it also reflects instrumentation, which is musical content. Each descriptor is binarized into its top and bottom tertile, with the middle dropped. This sidesteps the discrete-treatment limitation of classic matching at the cost of discarding data, which we report.
The second step estimates matched effects on MTG-Jamendo autotagging-moodtheme (The MTG-Jamendo dataset for automatic music tagging). For each descriptor $d$, a propensity model predicts membership in the high tertile from three groups of covariates. The first is the predicted mid-level perceptual features. The second is tempo, key and mode, and the genre tags. The third is an artist tagging-habit profile, computed only from that artist's other tracks, which absorbs the artist-as-annotator confound. Tracks are matched by nearest neighbor within a caliper, with balance checked by standardized mean differences. Overlap is reported explicitly, because genre conventions such as loud metal versus quiet folk will leave regions without support. The estimand is the effect on the treated, within the region of overlap.
For each tag $k$, the human effect $\tau^{H}_{d,k}$ is the matched mean difference in tag presence. The model effect $\tau^{M}_{d,k}$ is the matched mean difference in the model's score. Excess reliance is
$$ R_{d,k} = \tau^{M}_{d,k} - \tau^{H}_{d,k}, $$
where a model with $R = 0$ responds to production as the labels do. A plain invariance-trained model drives $\tau^{M}$ to zero and so has $R = -\tau^{H}$.
The third step validates the estimate with a randomized listening study. We take roughly 120 excerpts, render each in three mastering variants that move one descriptor at a time, and collect per-listener ratings on valence, energy, and tension, plus a handful of Jamendo tags. This gives an interventional human effect free of confounding, which we compare with $\tau^{H}$. If the two disagree badly, hidden confounding dominates and the observational target cannot be used. That is a reportable negative result. Per-listener ratings also let us estimate how much listeners differ in their sensitivity to production, which is the subjectivity question in its own right.
The fourth step is calibrated training. Each training batch includes mastering-edited copies $T_d(x)$ of clips $x$. An extra loss penalizes the difference between the model's score change on that pair and the target $\tau^{H}_{d,k}$, weighted by how confident the matched estimate is:
$$ \mathcal{L} = \mathcal{L}_{\text{tag}} + \lambda \sum_{d,k} w_{d,k}\big(f_k(T_d(x)) - f_k(x) - \tau^{H}_{d,k}\big)^2 . $$
Here $f_k$ is the model's score for tag $k$, $w_{d,k}$ is the inverse variance of the matched estimate, and $\lambda$ sets the strength of the term. Setting $\tau^{H} = 0$ recovers ordinary invariance augmentation, so that comparison is one line of code apart.
Baselines
The naive reference point is the popularity baseline on the artist-split MTG-Jamendo mood/theme task. MediaEval 2021 reports a macro PR-AUC of 0.031 for it. The organizers' 5-layer VGG-ish CNN reaches 0.107 on the same split. That is the minimum any trained model must clear, and it is the backbone we use for low-cost ablations.
The strongest existing method is a modern CNN tagger trained end to end. The specific architectures are the short-chunk CNN and Harmonic CNN, which The MTG-Jamendo dataset for automatic music tagging reports as top performers on MagnaTagATune and the Million Song Dataset. The file has no reported moodtheme number for those specific architectures. Challenges and Perspectives in Interpretable Music Auto-Tagging cites a best deep-learning ROC-AUC of 0.769 on MTG-Jamendo mood tagging, with its own XGBoost at 0.729. We treat 0.769 ROC-AUC as the bar for in-distribution accuracy. We retrain the Harmonic CNN ourselves under identical splits, epochs, and hyperparameter-search budget, and apply our objective on top of that same backbone. The interpretable XGBoost pipeline is a second strong reference for the diagnostic claim, because SHAP auditing is the existing way to find production shortcuts.
The key ablation is invariance augmentation: the same edits $T_d$ and the same loss with the target set to zero. This isolates the core ingredient, which is the human-estimated target rather than the augmentation. A second ablation replaces matching with plain regression adjustment when estimating $\tau^{H}$. A causal baseline adapts the context-pool backdoor adjustment of Xin et al. to the tagger. The file reports no result on Lacuna for it on MTG-Jamendo.
Beating the field requires three things together. On the artist split, PR-AUC and ROC-AUC must stay within noise of the strongest backbone, at about 0.769 ROC-AUC. Mean $|R_{d,k}|$ must fall clearly below both the unregularized and invariance-trained models. And on the externally rated sets, the models must track human ratings more closely when production shifts.
Why not just…?
Why not just use artist- and album-controlled splits? Those splits are already standard, and the 0.107 PR-AUC baseline already uses one. They remove leakage within a single artist. They leave genre-wide mixing conventions intact, and those conventions recur across artists on both sides of the split. A model can exploit them and still score well on the split. Our matched contrast is computed on that same held-out split. If artist splitting were sufficient, the unregularized baseline would already show $R \approx 0$, and we will report whether it does.
Why not just augment with compression, EQ, and gain to make the model invariant? This is the most natural alternative and the one our core ablation tests. Its weakness is conceptual. It sets the model's production effect to zero even where listeners respond to production. The MTG-Jamendo dataset for automatic music tagging also found that taggers respond very unevenly to different deformations, so uniform augmentation is a blunt instrument. If the listening study finds $\tau^{H}$ near zero for every descriptor, then invariance is the correct target and our method reduces to it. In that case we would say so and recommend the simpler method. The proposal is only worth more if the human effects are nonzero for some tags.
Why not just add production descriptors as covariates in a regression instead of matching? With a few covariates and roughly linear effects, regression adjustment may give the same $\tau^{H}$. We therefore run it, along with a doubly robust estimator, side by side. Matching is kept as the primary estimator because it forces an explicit check of overlap. It also produces pairs of tracks that a person can listen to. And unlike regression, it does not extrapolate into genre regions where no loud folk or quiet metal exists. If all estimators agree, the contribution lies in the estimand and the training target, not in matching itself, and we will present it that way.
Why not just train an interpretable model and audit it with SHAP? Lyberatos et al. did exactly this. SHAP showed reliance on spectral flux and similar cues, but it cannot say whether that reliance exceeds what listeners show. It also costs accuracy: 0.729 versus 0.769 ROC-AUC. We will report SHAP rankings next to $R_{d,k}$ for the same XGBoost model. The test is whether features flagged by SHAP are the ones with large excess reliance, or whether some of them reflect production effects that humans actually perceive.
Proposed Experiments and Decisive Checks
The first decisive check compares the observational and interventional human estimates. For each descriptor and rated dimension, we compare $\tau^{H}$ from matched Jamendo pairs with the effect measured in the randomized re-mastering study. We report rank correlation and sign agreement. Calibrated training is only justified for descriptors where the two agree.
The second check is the diagnostic itself. We compute $R_{d,k}$ for the VGG-ish model, the Harmonic CNN, XGBoost, the invariance-trained model, and the calibrated model on the MTG-Jamendo artist split. We report bootstrap intervals over matched pairs, along with the fraction of tracks trimmed for lack of overlap.
The third check covers accuracy and transfer. We measure PR-AUC and ROC-AUC on the artist split. We then run zero-shot transfer, using predictions mapped onto valence, tension, and energy. The targets are two externally rated sets: the Soundtracks set from Towards Explainable Music Emotion Recognition, where the A2E baseline reaches r = 0.76, and the per-rater MediaEval 2013 emotion annotations. On both sets we measure correlation with individual raters, not only with the mean. Finally, the held-out re-mastered excerpts serve as a production-shift test set.
The outcome that would refute the proposal is specific. Suppose the invariance-trained model matches the calibrated model on transfer and on agreement with listeners for re-mastered audio. Then the human production effect is either negligible or not estimable from observational tags. In that case the honest recommendation is plain invariance augmentation, with the matched contrast kept only as an auditing tool.
References
- The MediaEval 2013 Brave New Task: Emotion in Music
- MediaEval 2021: Emotion and Theme Recognition in Music Using Jamendo
- FMA: A Dataset for Music Analysis
- Challenges and Perspectives in Interpretable Music Auto-Tagging Using Perceptual Features
- Music Artist Classification with Convolutional Recurrent Neural Networks
- A Data-Driven Analysis of Robust Automatic Piano Transcription
- Improving Generalizability of Fake News Detection Methods using Propensity Score Matching
- Reasoning about Political Bias in Content Moderation
- A Data-driven Approach to Mid-level Perceptual Musical Feature Modeling
- Towards Explainable Music Emotion Recognition: The Route via Mid-level Features
- Improving Weakly Supervised Sound Event Detection with Causal Intervention
- Addressing The Confounds Of Accompaniments In Singer Identification
- The MTG-Jamendo dataset for automatic music tagging