Audio models trained mostly on Western music lose much of their accuracy on other traditions. One contrastive music model scores about 80% ROC-AUC on Western tagging and about 61% on Carnatic (South Indian classical) music, where melody lives in continuous pitch glides rather than discrete semitones. The usual fix is continual pre-training: keep training the Western model on non-Western audio, and mix in some Western data so it does not forget. That works modestly, but the risk is built in. The same weights must serve both repertoires, the mixing ratio is chosen by hand, and nothing guarantees the adapted model beats a small model trained on the target tradition alone.
We propose a different arrangement. The Western foundation model stays frozen. A small second encoder is pre-trained on unlabeled non-Western audio with an objective that only rewards it for predicting what the frozen model's features cannot. At downstream time, a shallow head reads both. Western performance cannot regress, because the Western model is never touched. Theory of negative transfer, the case where using a pre-trained model hurts compared with training from scratch, predicts where the second encoder should matter most: traditions far from Western music, and settings with more labeled target data. Together these give a concrete, falsifiable account of when cultural transfer helps and when it gets in the way.
Core Claim
The claim is that self-supervised pre-training of a residual culture encoder, conditioned on a frozen Western backbone, matches or beats two-stage continual pre-training on non-Western tagging. It does so at zero Western regression. Across label budgets, it also never falls below a culture-only model of the same size.
Pre-training the residual encoder on the backbone's errors, rather than independently, is what makes it more than an ensemble. Making its prediction targets resolve pitch more finely than the Western audio tokenizer addresses the "tokenizer bottleneck" that the closest prior work names as its main limitation.
Positioning Against Prior Work
The motivating evidence comes from two diagnostic studies. Universal Music Representations? Evaluating Foundation Models on World Music Corpora benchmarks audio foundation models on six roughly 200-hour corpora: Western, Greek folk, Turkish classical, Hindustani and Carnatic. It finds performance falls with cultural distance from Western music. It also finds that shallow fine-tuning barely beats a linear probe, which suggests the backbone's features set the ceiling. Do LLMs Distinguish Between Halal and Haram? shows the text analogue: small models pre-trained on native cultural corpora beat larger general ones (NileChat 3B F1 0.7707 vs Llama-3.1 8B F1 0.7268). Both papers diagnose the problem; neither proposes an architecture to fix it.
The closest method is CultureMERT. It continually pre-trains MERT-v1 on a 650-hour multicultural corpus in two stages, with 20% Western replay and a learning-rate re-warm, and also tests task arithmetic, which merges culture-specific weight differences back into the base model. It reports a 4.9% average gain on non-Western tasks with a 0.05% Western drop. It also reports that audio-token similarity predicts which traditions transfer to which, for example Turkish-makam to Carnatic. Our proposal differs in one structural way: CultureMERT rewrites the shared weights, while we never modify them and put all cultural learning in a separate, residual pathway.
The method we borrow is Residual Feature Integration is Sufficient to Prevent Negative Transfer (REFINE). It concatenates a frozen pre-trained representation with a trainable encoder learned from scratch on labeled target data. Under squared loss, it proves the resulting error adapts to whichever is faster: the source-aided rate or the from-scratch rate. REFINE's second encoder is trained only on downstream labels and was never tested on audio or culture. We move the residual idea into self-supervised pre-training, where unlabeled non-Western audio is relatively plentiful and labels are scarce.
REFINE's guarantee is for squared loss, and we use multi-label cross-entropy. We therefore treat "never worse than the culture-only model" as an empirical prediction to test, not an inherited theorem.
The theory that shapes our predictions comes from two analyses. Phase Transitions in Transfer Learning for High-Dimensional Perceptrons shows that transfer turns harmful below a critical source–target similarity. That threshold rises as target sample size grows. It also shows that soft transfer, which keeps target parameters close to source parameters with a penalty, beats freezing them. Transfer Learning in Infinite Width Feature Learning Networks shows that very strong source feature learning collapses representations onto the source task and destroys plasticity. That offers a candidate explanation for MERT-95M beating MERT-330M on world music.
Both results are proved for idealized models: Gaussian inputs, perceptrons and two-layer networks. We use them only to generate qualitative, checkable predictions about real systems. We do not claim they transfer exactly.
Research Program
The backbone $f$ is MERT-v1, frozen, with layer-averaged features $z = f(\tilde{x})$ computed on a masked audio input $\tilde{x}$. The residual encoder $h$ is a small transformer of roughly 30–95M parameters. It reads the same masked audio and receives $z$ through cross-attention, with gradients stopped into $f$. A prediction head $g$ is trained jointly with $h$ on the masked frames:
$$\mathcal{L} = \mathbb{E}_{x \sim D_{\text{culture}}} \sum_{t \in M} \ell\big(g([z_t;\, h(\tilde{x}, z)_t]),\; y_t\big)$$
Here $D_{\text{culture}}$ is the unlabeled non-Western audio, $M$ is the set of masked frames, $y_t$ is the pre-training target at frame $t$, and $\ell$ is cross-entropy. Because $z$ is fixed and already carries what the Western model knows, $h$ only receives useful gradient on the information $z$ is missing. That is the sense in which it is "residual."
We test two target families for $y_t$. The first is MERT's own discrete audio-codec tokens. The second adds cluster labels computed from pitch-salience features at sub-semitone resolution, along with fundamental-frequency contours. These are meant to capture gamakas (Carnatic pitch ornaments) and makam microtones that a Western-trained codec flattens.
For downstream use, a shallow head reads $[f(x); h(x)]$, the same fusion REFINE uses. We train three residual encoders. Two are per-tradition, for Turkish-makam and for the Carnatic and Hindustani traditions together. The third is a single multicultural encoder. This lets us test whether a "hub" tradition such as Turkish-makam yields a residual that helps others, as CultureMERT's token-similarity finding suggests.
Baselines
The naive reference points are two. The first is a linear probe on frozen MERT-v1. The second is a culture-only model: the same small transformer as $h$, pre-trained with the same objective but without access to $z$, then probed alone. The second one matters most to the theory. Under the phase-transition analysis, it is the model that the source-aided methods should eventually lose to on distant traditions as labels grow.
The strongest existing method is CultureMERT's two-stage continual pre-training with 20% Western replay, together with its task-arithmetic variant. We re-run both on CultureMERT's corpus and splits so the comparison is like for like. The reported figures to beat are a 4.9% average non-Western gain and a 0.05% Western drop for the two-stage model, and +0.4% on Western data for task arithmetic, with non-Western performance on par with the two-stage model (CultureMERT). The file gives no reported per-corpus absolute ROC-AUC for CultureMERT on Lyra, Turkish-makam, Hindustani or Carnatic.
As an external ceiling, we report that Qwen2-Audio reaches a mean ROC-AUC of 88.98% across the world-music corpora (Universal Music Representations?). It is a much larger model, so we report it only for context.
Fairness is enforced in three ways. Every pre-training arm receives the same unlabeled audio and the same GPU-hour budget; the residual arms update fewer parameters, so they get equal or fewer optimizer steps. Every arm is evaluated with the same probe and shallow-head protocol and the same hyperparameter-search budget. We report ROC-AUC and mAP over at least three seeds.
Two ablations isolate the key ingredient. The first trains $h$ independently and then concatenates it with $f$ only at downstream time. This is late fusion, or REFINE with a pre-trained second branch, and it tests whether conditioning on $z$ during pre-training matters. The second swaps the pitch-aware targets for codec-only targets, which tests whether the tokenizer bottleneck is a real cost.
We count the proposal as beating the strongest baseline if three things hold. It must match or exceed the re-run two-stage continual pre-training on the non-Western average, with a margin larger than the seed spread. It must show exactly zero Western regression. And it must stay at or above the culture-only model at every label budget.
Why not just…?
Why not just use CultureMERT's continual pre-training with replay? This is the real threat. A 4.9% gain at a 0.05% Western cost already leaves little room on the headline number. If our method only ties it, what remains is modularity: a culture can be added without touching or re-validating the backbone, at lower compute. The more substantive difference would be safety, and the label-budget sweep settles it. The phase-transition result predicts that a rewritten backbone can fall below the culture-only model on distant traditions once labels are plentiful. If continual pre-training never does, the safety argument fails and we will say so.
Why not just concatenate MERT with a separately pre-trained small culture model? That is our first ablation. If late fusion equals the conditioned version, then conditioning on the backbone's features adds nothing. In that case the contribution shrinks to "apply REFINE with a self-supervised second branch," which is still untested on audio but much weaker as a claim.
Why not just apply LoRA or fine-tuning to the general model? The world-music benchmark found that shallow fine-tuning barely beats probing (Universal Music Representations?). A code-generation study found that whether the base model saw domain data predicts results better than the choice of fine-tuning method (A Comparative Study of PEFT Methods for Python Code Generation). We still include LoRA continual pre-training of MERT, at the same compute budget, as an extra arm rather than assume it loses.
Why not just use a bigger general model such as Qwen2-Audio? It is strong, at a mean ROC-AUC of 88.98%, but it is roughly 25 times larger than MERT-330M. Its gains also come at a scale most cultural archives cannot train or serve. The question here is what a fixed, modest budget buys.
Proposed Experiments
The main experiment trains every arm on the CultureMERT corpus: MagnaTagATune and FMA-medium for Western music, plus Lyra, Turkish-makam, Hindustani and Carnatic. We evaluate with the probing and shallow-fine-tuning protocol of the world-music benchmark, reporting per-corpus ROC-AUC and mAP. We also run its 3-shot label-combination prototype-network protocol, since few-shot performance collapsed there and is where a residual pathway should help most.
The first decisive check is a label-budget sweep at 5%, 25% and 100% of the labeled training data for each tradition. For each point we plot the frozen probe, continual pre-training, task arithmetic, culture-only and residual arms. The phase-transition theory predicts three things. The gap between source-aided methods and culture-only should shrink, and may reverse, as labels grow, and do so faster for Carnatic than for Lyra. The residual arm should stay at or above culture-only. And the fusion head's reliance on $h$, measured by dropping each branch, should grow with label budget and cultural distance.
The second check plots the residual gain over continual pre-training against CultureMERT-style audio-token similarity to Western music. The prediction is that the gain grows with distance. A flat curve would mean residual decoupling is not addressing the cultural gap specifically.
The third check is the target ablation on gamaka-heavy Carnatic and microtonal makam material. We pair it with a pitch-trajectory probe built on the Carnatic melody resources from Repertoire-Specific Vocal Pitch Data Generation for Improved Melodic Analysis of Carnatic Music. This tests whether pitch-aware targets are what close the gap. Finally, we run every arm with both MERT-95M and MERT-330M backbones. The infinite-width theory reads MERT-95M's advantage as over-specialized larger features, and if that reading is right, the residual encoder should recover more of the gap for the larger backbone.
References
- Universal Music Representations? Evaluating Foundation Models on World Music Corpora
- Do LLMs Distinguish Between Halal and Haram? Benchmarking Islamic Cultural Alignment in General vs. Arabic-Centric SLMs
- CultureMERT: Continual Pre-Training for Cross-Cultural Music Representation Learning
- Residual Feature Integration is Sufficient to Prevent Negative Transfer
- Phase Transitions in Transfer Learning for High-Dimensional Perceptrons
- Transfer Learning in Infinite Width Feature Learning Networks
- A Comparative Study of PEFT Methods for Python Code Generation
- Repertoire-Specific Vocal Pitch Data Generation for Improved Melodic Analysis of Carnatic Music