Measuring Where Audio Metrics Go Deaf With an Adaptive Listening Game

Automatic audio metrics often disagree with human listeners, and we rarely know where. A metric may punish a tiny timing shift that nobody can hear, or ignore drum hits that fall off the beat, which everyone hears at once. This proposal builds a listening game that adjusts its difficulty to each player. As a player gets better at spotting controlled flaws in music, the game makes the flaws subtler, so every player ends up trained near the edge of what they can hear. That edge is the quantity we want. Across many players it gives a human detection threshold for each kind of flaw. Comparing those thresholds with how strongly each metric reacts to the same flaws gives a divergence profile: a per-flaw map of where a metric is oversensitive (it reacts to inaudible changes) and where it is deaf (it misses changes people hear). We then use the thresholds to recalibrate a learned music-quality model, and we test whether this improves agreement with expert judgments more per human label than ordinary preference collection does.

The opportunity comes from a coincidence between two fields. Adaptive gamification already keeps players at a challenge level matched to their skill, using models fitted to passive gameplay records. Psychophysics already estimates perceptual thresholds by adapting stimulus difficulty to each listener. Both are adaptive loops over a per-person skill estimate. In a game built this way, the difficulty curve and the threshold measurement are the same mechanism. The game earns its place only if it keeps people listening longer and more carefully than a plain test does, and we design the study so that this claim can fail.

Core Claim

Metric divergence in generative audio has structure along specific perceptual axes. Prior work shows the overall disagreement but not its shape. A Benchmarking Initiative for Audio-domain Music Generation using the FreeSound Loop Dataset argues that distribution metrics such as Fréchet Audio Distance (FAD, a distance between embedding statistics of real and generated audio) track surface texture and are blind to meter. ViSQOL v3 notes that full-reference metrics treat the small phase and timing shifts of neural codecs as serious errors. Improving Automatic Jazz Melody Generation by Transfer Learning Techniques finds that the model with the best distribution-overlap score (0.7536 versus 0.7490) was rated lower by jazz experts (2.52 versus 2.85 out of 5), while casual listeners preferred it. These papers motivate the proposal. Each reports a single disagreement on a single system and offers no way to measure the disagreement along many axes or across listeners who differ in expertise.

We claim that an adaptive game can measure human detection thresholds per flaw type and per listener, using gameplay data alone, with no self-report. We also claim that those thresholds expose metric failures that correlation with mean opinion scores hides, and that a learned evaluator recalibrated on the thresholds generalizes better to expert preference than one trained on the same number of pairwise votes.

Research Program

Stimuli. We build controlled stimuli from the FreeSound Loop Dataset (2,608 one-bar drum loops normalized to 120 beats per minute, from the FSLD benchmark). We apply parameterized manipulations, each with a continuous magnitude $\Delta$. Coherence manipulations include onset jitter off the metrical grid, swapped kick and snare positions, and partial reversal of the bar. Fidelity manipulations include low-bitrate codec passes, sub-millisecond phase and timing shifts of the kind discussed in ViSQOL v3, and spectral smearing from vocoder resynthesis. A second stimulus set applies the subset of manipulations that transfer to full songs (codec, timing, structural splicing) to clips from SongEval. This tests whether loop-level findings carry over to song-level aesthetics.

Game. The game is a short-session rhythm and repair game. On each trial the player hears three clips and must spot the "broken" one, the standard odd-one-out format in psychophysics. Correct answers on manipulated trials have a known ground truth, so scoring needs no consensus between players. This differs from TagATune, whose points depend on agreement with another player and whose difficulty never changes. Difficulty follows a per-player, per-axis staircase that holds success near 75%. That rate is the usual target for threshold estimation, and it also matches the frustration-avoidance logic that Towards Automatic Personalized Content Generation for Platform Games and Making Racing Fun Through Player Modeling and Track Evolution built into their content generators. Between scored trials, unscored preference trials show pairs of real generated outputs. We choose these pairs because existing metrics disagree about them, and we weight each player's votes by their measured acuity.

Listener model. For player $i$ and manipulation axis $k$, we fit

$$P(\text{correct} \mid \Delta) = \tfrac{1}{3} + \tfrac{2}{3}\,\sigma\big(a_{ik}(\Delta - t_{ik})\big),$$

where $\sigma$ is the logistic function, $t_{ik}$ is the player's threshold on that axis, $a_{ik}$ is the steepness of their response, and $1/3$ is the guessing rate for three choices. A hierarchical prior pools these parameters across players, so a short session still gives a usable estimate. Each player's vector of thresholds summarizes their listening skill. We test whether it separates self-described musicians from non-musicians, which would recover in a measurable form the expert-versus-casual split seen in the jazz study.

Divergence profile. For each metric $M$ (FAD, ViSQOL, Audiobox-Aesthetic, the SongEval-trained UTMOS model, and CMI-RM) and each axis $k$, we compute the metric's response at the population threshold $\bar t_k$. We divide that response by the metric's own variation across unmanipulated clips, which we call its noise floor. A ratio well below 1 means the metric is deaf to a flaw people hear. A large ratio for a change people cannot hear means the metric is oversensitive.

Recalibration. We fine-tune the SongEval UTMOS-style evaluator with an extra loss that has two terms. One penalizes score changes for manipulations below $\bar t_k$. The other requires a score drop of at least a margin for manipulations above $\bar t_k$. The acuity-weighted preference votes from the game supply a standard pairwise ranking loss alongside it. This builds on MusicRL, which trained a reward model on 300,000 unweighted, passively collected A/B votes. Our version uses labels that carry a known magnitude and come from listeners of known skill.

Baselines

The naive reference point is the set of off-the-shelf metrics used unchanged. For song aesthetics, SongEval reports that Audiobox-Aesthetic reaches Pearson correlation of 0.61–0.66 with human ratings. For FAD and ViSQOL thresholds on these manipulations, no reported result was found on Lacuna, so we compute them ourselves on identical stimuli.

The strongest existing evaluators are the UTMOS-based model trained on SongEval, which reports Pearson $r = 0.912$ with human averages, and CMI-RM from CMI-RewardBench. CMI-RM reports 78.20% accuracy on the CMI-Pref musicality task and 73.43% on Music Arena, against 65.80% on CMI-Pref for Gemini 3 Pro. We retrain the UTMOS-style model with the same architecture, data splits, and tuning budget, so that the only change in our version is the threshold and acuity-weighted data. Beating it means improving accuracy on held-out expert pairs from CMI-Pref and Music Arena without lowering SongEval correlation. A lower SongEval correlation would indicate the model is trading general quality for manipulation-specific behavior. CMI-RM is evaluated as released.

Data-collection baselines. Three comparisons use the same participant-hours. The first is a static web A/B test in the style of MusicRL, with randomly chosen pairs and no skill weighting. The second is a plain psychophysics staircase that uses the same adaptive algorithm with no game elements. This is the ablation that isolates the game itself. The third is a non-adaptive game with fixed manipulation levels, which isolates the adaptation. The game counts as beating these if it delivers more valid labels per participant-hour, longer voluntary sessions, and threshold estimates that are tighter (narrower posteriors) at equal trial counts.

Why not just…?

Why not just run a standard crowdsourced listening test? It can estimate an average preference. With fixed stimulus levels, though, most trials are either trivially easy or impossible for a given listener, so it wastes labels and measures no thresholds. It also yields no per-listener skill estimate, which is what the jazz result shows is needed. The static A/B baseline tests this directly at equal participant-hours.

Why not just use a psychophysics staircase without the game? This is the real threat. Adaptive threshold measurement already exists and may perform as well per trial. If the plain staircase matches the game on threshold precision and retention, we drop the claim that gamification helps. The divergence profile and the recalibration remain the contribution. We also test the concern raised by Beyond the Listening Test, which found that differences in perceived quality shrank when speech was heard inside an interaction. We compare thresholds from the game and from the plain staircase. A systematic shift would show that game context changes perception and would have to be reported, not hidden.

Why not just ask a multimodal language model to judge the audio? Gemini 3 Pro reaches only 65.80% on CMI-Pref musicality, and CMI-RewardBench reports that expert fine-tuning moves decisions away from model-generated labels. We include such a judge as a baseline on the divergence pairs. We expect it to show its own pattern of deafness, and we can measure that pattern with the same profile.

Why not just collect more preference votes and fine-tune the reward model, as MusicRL did? Votes alone show that people prefer one clip over another but not which perceptual axis drove the choice. They also mix skilled and unskilled listeners. CMI-RM's musicality and prompt-alignment scores remain correlated at Spearman ≈ 0.85, which suggests pooled preference data cannot pull those two judgments apart. Our matched-budget comparison between vote-only and threshold-plus-vote fine-tuning answers this question directly.

Decisive Checks

The proposal succeeds if three things hold. First, the divergence profiles show at least one coherence axis, such as meter jitter, where FAD and Audiobox-Aesthetic are deaf at human threshold, and at least one fidelity axis, such as phase shift, where full-reference metrics are oversensitive. Together these would confirm the earlier qualitative claims with measured magnitudes. Second, gameplay-derived thresholds predict, on held-out participants, whether a listener's preferences match expert labels on CMI-Pref. Third, the recalibrated evaluator beats the vote-only evaluator trained at equal label budget on held-out expert pairs, and holds SongEval correlation near 0.912. The proposal fails if the recalibration helps only on the manipulation types it was trained on. To test this, we hold out entire manipulation axes during training and evaluate on them.