Abstaining Active Learning for Music Preference Models

Automatic scores for generated music routinely disagree with what listeners hear. Signal-matching measures punish harmless timing shifts, distribution-level scores reward rhythmically incoherent drums, and even specialized learned reward models agree with experts only about three-quarters of the time. The standard fix is to collect expert ratings and train a predictor on them. That is expensive: one benchmark used 14 conservatory professionals for 13,740 ratings, another used 16 musicians listening to more than 140 hours of songs. It also hides a problem. For some pairs of clips, experts genuinely do not agree which is better, and no number of extra labels will settle it. A labeling strategy that keeps requesting judgments wherever the model is unsure will spend much of its budget on exactly these pairs.

This note proposes an active labeling procedure for music preference models built on one idea from learning theory: a learner does not need to know how noisy its labels are if it can detect noisy regions from the labels themselves and then stop asking about them. Concretely, the system queries experts on pairs of generated clips. It runs a sequential statistical test that separates pairs where more labels will resolve the ordering from pairs that are truly ambiguous. The reward model it trains outputs "no preference" on the ambiguous region instead of guessing. The expected payoff is fewer expert labels for a given ranking accuracy on the pairs that can be decided, and a reward model that knows when its preferences should not be trusted during best-of-N selection.

Core Claim

Disagreement among listeners about generated music is heteroscedastic: it varies from clip to clip and depends on context. The same crackle is a feature in lo-fi hip hop and a defect in classical piano, as CMI-RewardBench points out. Our claim is that an active learner which treats this disagreement as something to detect, rather than as uncertainty to resolve, reaches a target accuracy on decidable pairs with substantially fewer expert queries than uncertainty-driven or passive collection. It also needs no prior estimate of noise level. The learning target is Chow's risk, which counts a wrong preference as a full error and a "no preference" answer as a smaller fixed cost $c<1/2$:

$$R_c(f) = P\big(f(a,b)\neq y,\ f(a,b)\neq *\big) + c\,P\big(f(a,b)=*\big),$$

Here $f(a,b)$ is the model's verdict on clips $a$ and $b$, $y$ is a randomly drawn expert's preference, and $*$ is abstention. Under this objective the best predictor abstains exactly where experts are near a coin flip. Exponential savings in agnostic active learning through abstention shows that this objective lets active learning escape the usual $O(1/\epsilon^2)$ agnostic lower bound and reach $O(\log^2(1/\epsilon))$ labels. That is the theoretical reason to expect a large effect rather than a marginal one.

Existing work sits on either side of this idea. Maximizing Conditional Entropy for Batch-Mode Active Learning of Perceptual Metrics is the closest method. It actively selects diverse batches of perceptual triplets under a Gaussian model of response noise, and it was evaluated on taste, touch and image data. It never lets the learner decline to decide and never estimates where noise is high, so under our hypothesis it too would pour queries into ambiguous regions. The music benchmarks MusicEval, SongEval and CMI-RewardBench motivate the problem: all collect labels passively. MusicEval goes further and discards raters who disagree on sanity probes, which throws away the ambiguity signal we want to model.

Research Program

The hypothesis class is deliberately the one MusicEval already validated. It is a frozen CLAP encoder (an audio-text embedding model) with a small multilayer network on top, here applied to pairs of clips generated from the same prompt. We train a bootstrap ensemble of these heads. The explicit version spaces required by the abstention algorithm are intractable for neural models, as the file notes, and the ensemble stands in for them. A pair is a query candidate when the ensemble members disagree about which clip is better.

The noise-adaptive ingredient is borrowed from Active Learning from Imperfect Labelers. That work locates decision boundaries through on-the-fly significance tests rather than known noise rates. For each queried pair we keep a running count of expert preferences, including an explicit "can't decide" response wherever the data allow it. We apply an anytime-valid sign test with empirical-Bernstein confidence bounds. The pair is declared decided once the preference margin clears the bound. It is declared ambiguous once the bound shrinks below a tolerance tied to $c$ while still containing zero. No noise scale is set in advance, because the bound adapts to the observed variance.

Declared-ambiguous pairs leave the query pool. They also serve as positive examples for a separate abstention head that learns to predict ambiguity from audio. This lets the learner stop querying whole regions, such as a genre-and-artifact combination, rather than one pair at a time. That is the practical analogue of the region detection in Puchkin et al.

Batches are formed greedily. Among the eligible pairs, we pick those whose embeddings are least redundant with pairs already in the batch, reusing the conditional-entropy selection of Ebrahimi et al. as a diversity term. Batch diversity is not the contribution. Excluding ambiguous regions is.

We treat one constraint from Direction 1 as binding. A Polynomial-time Form of Robust Regression shows that jointly optimizing model parameters and per-point noise weights destroys convexity and traps optimizers. We therefore never fit a noise parameter jointly with the reward model. Ambiguity is decided by the external test, and the reward and abstention heads are trained on fixed targets.

Before any of this, one pre-check decides whether the project is worth running. Using the multiple ratings per clip in MusicEval (13,740 ratings over 2,748 clips, roughly five each) and SongEval, we measure whether rating variance differs substantially across clips and whether a held-out regressor can predict it from audio. If disagreement is roughly uniform, there is nothing to adapt to and the proposal reduces to ordinary active learning. The simulation also depends on per-rater scores being released. The file does not confirm this, so we will verify it first. If only means are available, we fall back to the live study described below.

Baselines

The naive reference point is random pair sampling at equal label budgets, training the same CLAP-plus-MLP ensemble. Its full-data endpoint is the MusicEval predictor, which reports a system-level Musical Impression SRCC of 0.926 and MSE of 0.446 (MusicEval). Our curves must reach an equivalent pairwise accuracy at a fraction of the labels. On SongEval we will also report correlation of the resulting scores against the UTMOS-based predictor's 0.912 Pearson correlation from SongEval, treated as a full-data ceiling.

The strongest existing reward model on this question is CMI-RM, which reports 78.20% accuracy on the CMI-Pref musicality task and 73.43% on Music Arena (CMI-RewardBench). It was trained on 110,000 pseudo-labeled pairs followed by 4,027 expert pairs. The fair comparison is therefore not our small model against their full pipeline. Instead, we take the same two-tower architecture pretrained on the same pseudo-labels, then fine-tune it on actively chosen subsets of CMI-Pref at 10%, 25% and 50% of the expert pairs. The same initialization is fine-tuned on random subsets under an identical tuning budget. Beating it means matching the 78.20% full-data accuracy with at most half the expert pairs, or exceeding it at equal budget on the pairs the model does not abstain on, with abstention cost charged via $R_c$.

The strongest active-learning competitors are three. The first is top-K ensemble-uncertainty sampling. The second is BALD-style mutual-information sampling, which uses the ensemble to separate model uncertainty from label noise in principle. The third is the conditional-entropy batch method of Ebrahimi et al. For all three, no reported result was found on Lacuna for music or audio preference data, so all are run by us with the same encoder, ensemble size, batch size and label budget. We also include the "objective metrics as disagreeing experts" sampler suggested by the selective-sampling view of Adaptive Selective Sampling for Online Prediction with Experts. It queries pairs where CLAP score, FAD-style embedding distance and mel distance disagree.

Two ablations isolate the key ingredient. The first removes the sequential ambiguity test while keeping the abstaining output. The second keeps the test but forces a binary output. An oracle variant that is told each pair's true noise level bounds how much the parameter-free test loses.

Why not just…?

Why not just use a zero-shot foundation model such as Audiobox-Aesthetic or a multimodal LLM judge? The file's evidence is direct. Audiobox-Aesthetic reaches only 0.61–0.66 Pearson correlation with expert song ratings, against 0.912 for a model trained on human labels (SongEval). Gemini 3 Pro reaches 65.80% on CMI-Pref musicality (CMI-RewardBench). Human labels remain necessary, so the real question is how to spend them. We will still include the zero-shot judge as a floor.

Why not just collect more ratings passively and filter noisy raters? This is the MusicEval recipe, and it works if one can afford it. Filtering, though, treats disagreement as rater error. Our pre-check measures directly whether disagreement concentrates on predictable clips. If it does, filtering removes exactly the information the abstention head needs. The random-sampling curve in our experiments is this alternative at every budget, so the cost argument will be settled by numbers rather than asserted.

Why not just use BALD, since ensembles already separate reducible from irreducible uncertainty? This is the real threat. BALD down-weights pairs where every ensemble member predicts about 50/50. With few labels, however, members fit noisy pairs differently and disagree, so BALD can still chase ambiguity until enough repeats accumulate. Our explicit test is designed to catch this sooner. We will know which is right from the BALD curve against ours. We will also run BALD equipped with our abstaining output. If that combination matches the full method, the honest conclusion is that the abstaining objective, not the sequential test, carries the gain, and we will report it that way.

Why not just query where the objective metrics disagree? It is cheap and needs no trained model. Nothing in the file, however, shows that disagreement among signal-level metrics tracks disagreement among human listeners. Hung et al. show distribution metrics miss rhythm entirely (A Benchmarking Initiative for Audio-domain Music Generation using the FreeSound Loop Dataset), so metric clashes may mostly flag texture differences that experts consider irrelevant. It runs as a baseline, and we will report the correlation between metric disagreement and measured human rating variance.

Proposed Experiments

The primary experiment is a pool-based simulation on MusicEval and SongEval, covering Musical Impression and Overall Musicality. It uses same-prompt clip pairs whose ground-truth label is the sign of the mean expert rating difference. Pairs whose mean difference falls inside the pooled rating noise are marked truly ambiguous. Each query returns one stored rater's judgment, drawn without replacement, and a tie on the Likert scale counts as "can't decide." Repeat queries of the same pair are allowed until its raters are exhausted. We report three quantities against label count, each over ten seeds: excess Chow's risk at $c\in\{0.2,0.3,0.4\}$, accuracy on truly decidable pairs, and the fraction of queries spent on truly ambiguous pairs. That last quantity is the mechanism check, since the method should spend far less there than uncertainty sampling.

The secondary experiment is the CMI-Pref subset study described under Baselines. It tests whether the query rule helps even without repeated raters.

The third experiment is best-of-N reranking with $N=10$, matching the CMI-RewardBench protocol, on held-out MusicEval prompts. When the reward model abstains among the top candidates, it defers to a tie-break (prefer the candidate with fewer predicted-ambiguous comparisons). The measure is the held-out expert mean rating of the selected clip.

A small live study closes the loop. It uses a handful of experts, pairs from current text-to-music generators, and an explicit "no preference" button. The study tests whether real abstention rates peak where the simulated test says they should.

The decisive checks are three, stated in advance. First, the heteroscedasticity pre-check must show predictable per-clip variance, or the project stops. Second, the method must reach the random-sampling full-data decidable-pair accuracy with no more than half the labels, and must beat BALD equipped with abstention at equal budget. Otherwise the contribution reduces to the abstaining objective alone. Third, abstention-aware reranking must not select lower-rated clips than CMI-style scalar reranking. If it does, abstention is costing more than it saves.