Does Hardware-Friendly Sparsity Widen the Cultural Gap in Music Models?

Audio foundation models for music already do much worse on non-Western traditions than on Western pop. Their quality falls as the music moves further from the Western training data. One audit finds a CLAP music model's tagging score drops from roughly 80% ROC-AUC on Western corpora to roughly 61% on Carnatic music (Universal Music Representations?). Deploying these models cheaply means compressing them. The only form of weight sparsity that reliably speeds up a modern GPU is "2:4" sparsity, a pattern where exactly two of every four consecutive weights are zero, which NVIDIA's sparse tensor cores can run at up to twice the math throughput. Arbitrary ("unstructured") zeros mostly do not pay off in wall-clock time. In vision, pruning is known to hurt rare subgroups far more than the average (Selective Brain Damage; Pruning has a disparate impact on model accuracy). Nobody has checked whether the one sparsity pattern that actually speeds things up quietly makes music models worse at precisely the traditions they already handle badly.

This note proposes two things. The first is a measurement: 2:4 compression of music foundation models, with wall-clock speed actually measured, and disparate harm tracked per musical tradition. The second is a targeted repair. Current methods that make 2:4 pruning less damaging pair the sparse weights with a small dense correction, and they fit that correction to English text or to whatever calibration audio is at hand. We instead fit the correction under a min-max objective across musical traditions, so that no single tradition absorbs the compression error. The two-sentence version: compressing a Western-trained audio model to the GPU-friendly 2:4 pattern probably sacrifices Carnatic, makam and Greek folk performance first, because the calibration data and the error objective both favour the majority. We measure this, and we test whether a culture-balanced fit of the small dense correction recovers it at the same measured speed.

Core Claim

We make two falsifiable claims. First, under standard post-training 2:4 compression of MERT-style encoders, the relative drop in tagging quality grows with cultural distance from the Western pre-training data. Cultural distance here is measured by the acoustic-token similarity that CultureMERT found predicts cross-cultural transfer. If true, the vision findings carry over to audio and to hardware-realistic sparsity. Second, most of this extra harm is fixable without giving up the speedup. The fix is to fit the dense correction under a worst-tradition reconstruction objective, rather than only to reweight calibration data or to fine-tune afterwards.

The second claim is built so that it can fail cleanly. If simply mixing multicultural audio into the calibration set closes the gap just as well, the min-max objective adds nothing, and we will report the result as a calibration-data finding.

Research Program

The method comes from sparse-plus-low-rank compression. ARMOR wraps a 2:4 sparse core between two block-diagonal matrices fitted layer by layer, and cuts the 2:4 perplexity gap by up to half on LLaMA-2-13B. 3BASiL writes each weight matrix as a sparse part plus a low-rank part, solves for both jointly with an alternating optimizer, and adds a block-level refinement step. Both are calibrated on English text corpora, and both minimise a single pooled reconstruction error. We keep their machinery and change what is being fitted.

For a linear layer with dense weights $W$, let $X_c$ be that layer's inputs on calibration clips from tradition $c$ (Western, Lyra, Turkish-makam, Hindustani, Carnatic). We fit a 2:4-patterned sparse matrix $S$ and a rank-$r$ matrix $L$ by solving

$$ \min_{S \in \mathcal{S}_{2:4},\, \mathrm{rank}(L)\le r}\; \max_{c}\; \frac{\lVert W X_c - (S+L) X_c \rVert_F^2}{\lVert W X_c \rVert_F^2}, $$

where $\mathcal{S}_{2:4}$ is the set of matrices obeying the 2:4 pattern. Dividing by $\lVert W X_c \rVert_F^2$ makes the errors comparable across traditions whose activations differ in scale. We solve this in practice as a weighted sum. The weights $w_c$ are raised multiplicatively for whichever tradition currently has the largest relative error, inside the existing 3BASiL alternating updates.

This is the post-training, layer-wise analogue of the group-loss balancing in Pruning has a disparate impact on model accuracy. That work rebalances the training loss after unstructured pruning of vision models, and shrank the UTK-Face group gap from 0.134 to 0.073. We rebalance the compression objective itself, with no task labels, so the compressed encoder stays general-purpose.

Two design choices are fixed by prior evidence. The first concerns the convolutional front end, which turns raw audio into the low-level acoustic features the transformer reads. We leave it dense, because 2:4 acceleration applies only to matrix multiplications, and because CultureMERT found this front end is where non-Western adaptation must be stabilised. The second is the measurement of harm. Alongside top-line scores, we count "pruning-identified exemplars" in the sense of Selective Brain Damage: clips where the compressed model's tags disagree with those of an ensemble of dense models. We report these per tradition, because averages hid catastrophic local collapse in the vision setting.

We also keep in view a tension in the file. MERT-95M often beats MERT-330M on non-Western corpora (Universal Music Representations?), so compressing the larger model might shed some Western-specific capacity rather than cultural capacity. Our first experiment is therefore an honest audit. It could come out with no disparate harm, or even with gains, and that result would be reported as such.

Baselines

The naive reference point is the dense encoder: MERT-v1-330M, MERT-95M and the CultureMERT checkpoint, each with frozen-feature linear probes on each corpus. It sets the quality ceiling and the unaccelerated latency.

The second reference is magnitude-based 2:4 pruning with no correction term. It sets the speed ceiling that ARMOR calls the "naive" one: 1.86x against ARMOR's own 1.57x on Qwen 2.5-14B in PyTorch. It also sets the expected worst case for disparate harm.

The strongest existing method is 3BASiL with its block-level refinement, plus ARMOR, both run on audio exactly as published. They are the strongest competitors because they are the best-reported way to make 2:4 pruning accurate. On text, 3BASiL with 2:4 sparsity plus a rank-64 correction cuts the WikiText2 perplexity gap by more than 30% relative to earlier sparse-plus-low-rank methods on LLaMA-3-8B. No reported result found on Lacuna for any 2:4 or sparse-plus-low-rank method on audio encoders, and none for per-corpus MERT ROC-AUC values. We will therefore establish both ourselves.

The fairness of the comparison rests on three controls. Every compressed variant uses the same total calibration audio. Every variant has the same rank $r$ and wrapper size, so its added FLOPs fall within ARMOR's reported 3–5% overhead. Every variant gets the same number of solver iterations and the same probe-tuning budget.

The ablation that isolates our key ingredient is 3BASiL with the pooled objective, calibrated on the same culture-balanced audio mix. Against this ablation, a win for our method means the min-max objective, not the data, does the work.

Beating the baselines means the following. Our method must raise the worst-tradition ROC-AUC, and lower the spread in relative drop between Western and Carnatic. It must do so at equal measured latency, and without losing more than a fixed tolerance on Western MTAT and FMA-medium. That tolerance is anchored to CultureMERT's reported 0.05% Western regression.

Why not just…?

Why not just calibrate existing 2:4 methods on multicultural audio? This is the most serious threat, and it is built into the design as the key ablation rather than argued away. Pooled objectives weight each tradition by how much activation energy and data volume it contributes. Western audio dominates the corpora that exist, at more than 94% of training data according to Music for All. Even a balanced pool can therefore let the hardest tradition's error grow. If the balanced-pool ablation matches our min-max fit on worst-tradition scores, we drop the objective claim and publish the calibration result.

Why not just quantize to INT8 instead of pruning? Selective Brain Damage does find quantization less disparately harmful than unstructured pruning, so it is a real option. But it does not use the sparse tensor cores, and the two techniques stack. We run INT8 alone, and INT8 combined with each 2:4 variant, and place all of them on the same measured latency-versus-worst-tradition curve. That curve shows directly whether sparsity buys anything beyond quantization for this workload.

Why not just use the dense MERT-95M, which already beats the 330M model on world music? This is a strong alternative given Universal Music Representations?. A 2:4 version of the 330M model still does more arithmetic than dense 95M, so a win is not guaranteed. We compare at matched measured A100 throughput, and also compress 95M itself. If dense 95M dominates the compressed 330M model at equal speed, we will say that compression is the wrong lever for this model family.

Why not just prune and then re-adapt with CultureMERT-style task vectors or adapters? Task arithmetic and adapters (CultureMERT; Music for All) need labelled or tradition-specific training. Adapters also add dense compute. We run "naive 2:4 followed by task-vector merging of the dense CultureMERT delta, re-projected onto the mask" as an extra baseline. Re-projecting onto the mask is needed because adding a dense delta would otherwise break the 2:4 pattern.

Proposed Experiments

The evaluation uses the six roughly 200-hour corpora from Universal Music Representations?: MagnaTagATune and FMA-medium for Western music, plus Lyra, Turkish-makam, Hindustani and Carnatic. Metrics are frozen-probe ROC-AUC and mAP per corpus, the per-tradition pruning-identified-exemplar rate, and measured encoder throughput on an A100 with sparse tensor-core kernels. We report measured throughput rather than theoretical FLOPs.

Calibration clips are held out from the probe splits. Turkish-makam is held out of calibration in one condition, to test whether the fix transfers to an unseen tradition, given CultureMERT's finding that makam-trained models transfer to Carnatic.

Three checks decide the outcome. First, whether naive and pooled 2:4 compression produce a relative drop that grows with token-similarity distance. A flat or inverted trend falsifies the premise, and also speaks to the MERT-95M/330M tension. Second, whether the min-max fit beats the balanced-pool ablation on worst-tradition ROC-AUC. Third, where every method sits on the curve of measured latency against worst-tradition quality, against dense MERT-95M and against INT8.