Automatic judges of music quality now steer music generators. These include learned reward models, which score audio in place of a human listener, and classic signal metrics. When a judge is wrong, a generator trained against it learns the wrong thing. The documented failure has two faces. Judges punish deliberate stylistic deviations, such as a jazz chromatic approach, a sung glissando, or a guzheng pitch bend, as if they were mistakes. Judges also miss real mistakes in genres they rarely saw during training. Current benchmarks report one averaged agreement number per judge, so neither face is visible.
We propose to measure both faces directly, genre by genre. We build matched pairs of recordings: an original, a version containing a real stylistic deviation, and a version containing an injected defect. A good judge should score the deviation like the original and the defect clearly lower. We then test whether a cheap repair works. The repair trains a judge on these contrast pairs, using only a subset of genres, and checks whether it holds on unseen genres. The central number is the judge's worst-genre ability to tell artistry from error, not its average agreement.
The evaluation design borrows from a field outside machine learning. Dental debridement research does not ask whether a root canal is clean "on average." It asks which anatomical regions the instrument failed to reach. Comparison of cleaning efficiency and deformation characteristics of Twisted File and ProTaper rotary instruments found that neither rotary file system fully cleaned canal walls, even though one was much faster. The Periodontal Disease Associated with the Palato-Gingival Groove of Upper Incisor showed that a single narrow groove can remain a permanent reservoir of bacteria under otherwise successful treatment. We treat under-represented genres as the grooves of a music judge. We report the judge's residual error region by region, because an average hides exactly the places that matter. The dental work supplies this measurement philosophy only; it contributes no mechanism, and we do not claim otherwise.
Core Claim
Current music judges conflate "unusual" with "wrong." The conflation is worst in under-represented genres, and averaged benchmarks cannot detect it. Our claim has three parts.
First, a contrast-pair audit exposes large per-genre gaps between how judges treat legitimate deviations and how they treat real defects. These gaps are invisible in reported numbers such as SongEval correlation or Music Arena accuracy.
Second, a lightweight judge trained with a ranking objective on such pairs narrows the worst-genre gap. It does this without losing correlation with human aesthetic ratings.
Third, the gain survives when entire genres are held out of training. That result would mean the judge has learned a transferable notion of "defect," not a genre-specific lookup.
Why This Gap Is Real
The file offers three kinds of evidence for the problem.
The first concerns deviations being punished. Two papers show that signal and distribution metrics reward the wrong things. Improving Automatic Jazz Melody Generation by Transfer Learning Techniques found that expert listeners preferred models that scored worse on statistical divergence, because those metrics treat chromatic enclosures as errors. A Benchmarking Initiative for Audio-domain Music Generation using the FreeSound Loop Dataset showed that Fréchet Audio Distance, a distance between feature distributions of real and generated audio, rewards surface texture and is blind to meter. ViSQOL v3 is a perceptual metric that compares a signal against a clean reference. Its authors concede that it mis-scores generative codecs whose output sounds good but is not sample-aligned with the reference.
The second concerns uneven coverage across genres. The learned judges are better but uneven. CMI-RewardBench reports that its reward model does noticeably worse on non-Western and experimental styles and on non-English prompts. It also reports that the model's musicality and instruction-following scores are correlated at about 0.85, a sign that the judge cannot separate distinct qualities. CMI-Bench documents that audio-language models degrade on GuZheng_99 technique detection and hallucinate beat timestamps.
The third concerns what averaged evaluation hides. SongEval and MuseCritic report strong overall agreement with experts. However, they cover mainstream vocal songs in English and Mandarin only, and they report no per-genre breakdown of defect sensitivity.
The closest existing work is this pair of papers, SongEval and MuseCritic, together with CMI-RewardBench. All three train judges to match averaged human ratings on mainstream corpora. None constructs controlled pairs that separate legitimate deviation from defect. None stratifies judge failure by genre. None tests whether judges hold on held-out genres. Those three things are what we add.
Research Program
Building the contrast set. The legitimate-deviation arm comes from real, annotated performances wherever possible. GTSinger (GTSinger/GTSinger) is 80.59 hours of studio singing in nine languages. It carries phoneme-level labels for vibrato, glissando, falsetto, breathy, pharyngeal and mixed voice, plus MusicXML scores. This lets us pair technique-bearing segments with plain-sung segments by the same singer. GuZheng_99, as used in CMI-Bench, supplies annotated playing techniques for a non-Western plucked instrument. The Latin Music Database supplies syncopated dance genres curated by experts. GuitarSet and Bach10 cover plucked and orchestral instrumental music; these are the regimes where Timbre-Trap found that pitch and timbre remain entangled for guitar.
The defect arm uses targeted edits applied at score-aligned locations:
- off-key note substitutions;
- deleted or duplicated beats spliced into the audio;
- abrupt tempo discontinuities;
- short dropouts;
- mistimed entries of one voice.
Each edit is small in signal energy, so the defect and the legitimate deviation are matched in how far they move the waveform from the original.
Validating the labels. Some pairs need synthetic legitimate deviations, such as microtiming warps drawn from measured human timing ranges. A listening panel rates a sample of every edit type in every genre stratum. The panel includes practitioners of guzheng and Latin dance music, not only Western-trained annotators. Edit types that the panel does not reliably classify as "stylistic" versus "mistake" are dropped. This guards against baking our own Western assumptions into the labels.
The audit. Every judge scores every triple. The primary measure is a separation score. It is defined as the probability that the judge ranks the original above the defect, minus the probability that it ranks the original above the legitimate deviation. The measure is computed per genre stratum, and we report the worst stratum alongside the mean. A judge that punishes everything unusual scores near zero. A judge that tells artistry from error scores near one.
The repair. We start from the SongEval-trained UTMOS regressor and a frozen general audio encoder. On top of these we train a small scoring head with two terms:
- a margin ranking loss that pushes originals above defects;
- a tie loss that keeps legitimate deviations near their originals.
A regression term to SongEval ratings preserves agreement with human aesthetic judgments. We also include one borrowed ingredient as a tested option, not an assumption. The borrowed idea comes from Using Intermittent Synchronization to Compensate for Rhythmic Body Motion During Autonomous Surgical Cutting and Debridement. There, a surgical robot acted only at stable instants of periodic tissue motion rather than tracking continuously. This raised foreign-body removal success from 62% to 80% and cut error 2.6×.
Our analogue compares local windows anchored at detected beats. Between anchors, the judge tolerates timing drift rather than demanding continuous alignment. This targets the micro-timing penalty ViSQOL's authors describe. It is an ablation, and we expect it may matter only for rhythmic genres.
Baselines
Naive reference. The do-nothing reference is mel-spectrogram L1 distance to the original, together with ViSQOL v3 run as its authors released it. Both measure signal change. By construction they should score legitimate deviations and defects about equally, so they set the floor for separation. Any learned judge that fails to beat them is a signal metric in disguise.
Strongest scalar judge. The UTMOS regressor trained on SongEval is the strongest scalar judge in the file. SongEval reports Pearson 0.912 with human ratings, against 0.61–0.66 for Meta's Audiobox-Aesthetic. Audiobox-Aesthetic is included as the off-the-shelf reference.
Strongest pairwise judges. CMI-RM from CMI-RewardBench reports 78.20% on CMI-Pref musicality and 73.43% on Music Arena. MuseCritic is the strongest reasoning-based judge; it reports 71.35% on Music Arena. Gemini 3 Pro, at 65.80% on CMI-Pref musicality per CMI-RewardBench, represents a prompted frontier model.
For the contrast audit itself there is no reported result found on Lacuna for any of these judges. The audit baselines are therefore our own measurements.
Running the comparison fairly. All judges score identical clips at identical durations, capped at 120 seconds, the point where CMI-RewardBench reports accuracy gains level off. Released weights are used without tuning. For the repair, the fair competitor is the same UTMOS model fine-tuned on the same contrast pairs with plain regression to pseudo-labels. It gets matched compute and a matched hyperparameter search budget.
Ablations. Two ablations isolate what drives any gain. The first removes the tie loss, leaving only defect ranking. It tests whether protecting legitimate deviations matters, or whether defect sensitivity alone explains the result. The second removes beat anchoring and tests the synchronization idea.
What counts as beating the baselines. The repair must satisfy three conditions:
- worst-stratum separation clearly above UTMOS-SongEval and CMI-RM, with non-overlapping bootstrap intervals;
- SongEval correlation no more than a few hundredths below 0.912;
- Music Arena accuracy not below MuseCritic's 71.35%.
Why not just…?
Why not just fine-tune an existing reward model on more non-Western music? This is the real threat. Adding GuZheng_99 and Latin Music Database audio with human ratings might close the genre gap without any contrast pairs. However, averaged ratings carry no signal telling the model which specific feature was a flaw and which was artistry. The jazz result in Improving Automatic Jazz Melody Generation by Transfer Learning Techniques suggests distribution-matching objectives learn to penalize the rare-but-good. We test this directly with a matched-data condition: the same added audio, trained with plain regression versus contrast ranking. If plain regression matches our separation scores, the contribution reduces to the audit, and we will say so.
Why not just use a reasoning judge like MuseCritic or a frontier model? Critique-based judges may already explain away ornaments. However, MuseCritic was trained only on Chinese and English vocal songs and is slow to run. Frontier models trail specialists by a wide margin, per CMI-RewardBench and CMI-Bench. Both are audited on the same triples. If MuseCritic already achieves high worst-stratum separation, the audit has still shown something new: it identifies which judge to trust for which genre.
Why not just use a time-tolerant signal metric such as ViSQOL? Time alignment removes the micro-timing penalty. It cannot distinguish an intentional glissando from an off-key slide, because both move spectral energy by similar amounts. Our energy-matched design makes this measurable. ViSQOL's separation score is the direct answer.
Proposed Experiments and Decisive Checks
Audit. The first experiment runs every baseline on the validated contrast set across roughly six genre strata: Mandarin and English pop vocals from SongEval-style material, GTSinger's other languages, guzheng, Latin dance, guitar, and orchestral. We expect learned judges to show much larger worst-to-mean separation gaps than their headline benchmark numbers suggest. If the gaps are small everywhere, the premise is wrong. That would be a useful negative result, and the repair work would stop.
Leave-one-genre-out transfer. This is the decisive check. The repair is trained on all strata but one and evaluated on the held-out stratum, for each stratum in turn. Transfer to held-out guzheng and guitar, the two regimes where prior work reports failure, is the result that matters.
Human-rating preservation. The repair must maintain SongEval correlation and Music Arena accuracy against the numbers quoted above.
Downstream sanity check. A small generator is fine-tuned with GRPO, a reinforcement-learning method that updates the generator using the judge's scores, as MuseCritic did. We compare the original judge and the repaired judge as rewards. Blind listeners, including genre practitioners, then compare the two generators' outputs. A repaired judge should yield outputs that keep ornamentation, rather than smoothing it away as the generator chases the original judge's approval.
References
- Comparison of cleaning efficiency and deformation characteristics of Twisted File and ProTaper rotary instruments
- The Periodontal Disease Associated with the Palato-Gingival Groove of Upper Incisor
- Improving Automatic Jazz Melody Generation by Transfer Learning Techniques
- A Benchmarking Initiative for Audio-domain Music Generation using the FreeSound Loop Dataset
- ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metric
- CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction
- CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Following
- SongEval: A Benchmark Dataset for Song Aesthetics Evaluation
- MuseCritic: Learning Multi-Aspect Song Rewards through Natural-Language Aesthetic Critiques
- GTSinger: A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Tasks
- The Latin Music Database
- Timbre-Trap: A Low-Resource Framework for Instrument-Agnostic Music Transcription
- Using intermittent synchronization to compensate for rhythmic body motion during autonomous surgical cutting and debridement