Rater State as a Hidden Variable in Listening Tests

When people rate synthetic speech or AI-generated music, the score they give reflects more than the audio. It also reflects the listener at that moment: how tired they are, how harsh their standards have become after twenty bad clips, whether the previous clip was so good that this one sounds worse by contrast. Today's automatic quality predictors treat a listener as at most a fixed offset. Everything else about the listener gets averaged away as noise. We propose to treat each rater the way systems neuroscience now treats a sensory cortex. Their response to a fixed stimulus is shaped by an internal state that drifts over time. That state can be inferred from their own sequence of responses, and it can be separated from the part of the response the stimulus actually drives.

This matters because the field's main complaint about audio metrics is that they "diverge from human perception." Part of that divergence may not be a failure of the metric at all. It may be that the human target is contaminated by session-level drift that no model tries to remove. If so, removing it gives two practical payoffs. Predictions for new listeners improve after only a handful of calibration ratings. And a listening test needs fewer ratings to reach the system ranking that a large, carefully balanced panel would produce.

Core Claim

A listener's rating of clip $x_t$ at position $t$ in a session can be written as an audio-driven quality term plus a latent rater state $s_{j,t}$ that evolves across the session. The rating is then passed through an ordinal observation model suited to a 1–5 scale:

$$P(r_{j,t} \le k) = \sigma\big(c_k - q(x_t) - s_{j,t}\big), \qquad s_{j,t} = g(s_{j,t-1}, u_{j,t}) + \epsilon_t.$$

Here $r_{j,t}$ is rater $j$'s score, $c_k$ are learned cut points between adjacent scale values, $\sigma$ is the logistic function, and $q(x)$ is a quality network that sees only the audio. The state update $g$ is a small recurrent network driven by observable trial covariates $u_{j,t}$: position in the session, the previous rating, the predicted quality of the previous clip, response latency, and replay count. The noise term $\epsilon_t$ lets the state absorb fluctuations that the covariates do not explain.

The claim is that $s_{j,t}$ carries real, structured variance beyond a static per-rater offset and beyond simple order covariates. The test is that removing it improves the stability of $q$ under different subsets of raters.

The split mirrors work on mouse visual cortex. Modeling Dynamic Neural Activity by combining Naturalistic Video Stimuli and Stimulus-independent Latent Factors pairs a stimulus network with a recurrent latent encoder. Its unsupervised latents recovered pupil dilation (canonical correlation of about 0.60) and running speed (about 0.42) without being given them. Time-Evolving Dynamical System for Learning Latent Representations of Mouse Visual Neural Activity explicitly separates stimulus latents from internal latents and shows that shuffling time order destroys decoding. We reuse that structure. Clips play the role of stimuli, ratings play the role of responses, and the observable covariates play the role that pupil and locomotion play in Sinz et al. (2018).

The neuroscience also supplies a warning that motivates the work. Contribution of behavioural variability to representational drift and Sadeh (2022) show that apparent drift in responses to identical stimuli is largely unmodeled behavioral change. Averaging over it manufactures artifacts rather than revealing a clean signal.

What Is Already Done, and What Is Not

Three papers already condition quality prediction on the listener, and they are the direct predecessors.

MBNet splits prediction into a mean network and a bias network keyed by a one-hot judge ID. It raises system-level Spearman rank correlation (SRCC) on VCC 2018 from 0.922 to 0.949.

UTMOS adds static listener and domain embeddings to fine-tuned self-supervised speech encoders and won the VoiceMOS Challenge 2022 main track.

Using Rater and System Metadata to Explain Variance in the VoiceMOS Challenge 2022 Dataset fuses audio features with rater and system IDs, uses an "unknown" token for unseen raters, and itself recommends moving beyond fragile ID memorization.

All three model the listener as time-invariant. All three handle new listeners either not at all or with a generic fallback. All three fit scores with squared-error losses that ignore the bounded, ordinal nature of the scale.

Our proposal differs on each of these three points. The rater term changes within a session. It is inferred for a new rater from their first few responses, by amortized filtering rather than a lookup table. And it sits under an ordinal likelihood.

The proposal also differs in what it measures. None of the prior work tests whether removing rater state makes system rankings more reproducible, which is the property a benchmark actually needs.

Research Program

Data: existing logs and a new collection. The first component is data that records order, which existing public benchmarks may not provide. The research file does not establish that the VoiceMOS 2022 data (The VoiceMOS Challenge 2022) or SongEval release per-rater presentation order. We will first check, and use any order information that exists.

Independently, we will run a new listening study designed after the repeated-trial protocol of sensory neuroscience. About 120 raters will each complete sessions of roughly 100 clips. Speech clips come from the public VoiceMOS 2022 main-track systems. Music comes as 20–30 second excerpts from SongEval songs. Each session interleaves eight anchor clips, each presented three times at random positions. Order is randomized per rater, and latency and replays are logged.

The repeated anchors are the key design choice. Any change in a rater's score for the same anchor is, by construction, not about the audio. This gives a ground-truth handle on drift that observational data cannot provide.

Model: a nested hierarchy. The second component is the model, built as a nested sequence so that each ingredient can be removed. Starting from a self-supervised audio encoder for $q$, as in UTMOS, we add these terms one at a time:

  1. a static rater offset;
  2. a deterministic covariate-driven state;
  3. the full stochastic latent state;
  4. the ordinal likelihood in place of squared error.

For unseen raters, an encoder reads their first $k$ (rating, clip) pairs and outputs the initial state. This parallels how the latent-factor model above conditions on part of the neural population to infer state for the rest.

The model is parameter-heavy and could let the latent simply memorize ratings. The latent-factor paper flags exactly this risk for neural data. We therefore keep the state low-dimensional, regularize it with a variational bottleneck, and never let the state network see the audio of the current clip.

Analysis of what the latent captures. The third component asks what the inferred state contains. We will correlate it with session position, latency, and the quality of preceding clips. We will compare its magnitude across speech and music, and across regular and expressive material. The comparison with expressive material follows from SpeechJudge, which reports that agreement drops on expressive speech. The comparison with music follows from SongEval's long excerpts, where fatigue plausibly dominates.

Baselines

Naive reference. The do-nothing baseline is a mean-opinion regressor in the MOSNet style. It is the same audio encoder trained on per-clip averaged scores with no rater information. Its predictions double as the plain average-the-ratings estimate of system quality.

Strongest published competitor on public data. This is the UTMOS recipe: a fine-tuned self-supervised encoder with a static listener embedding, plus the contrastive ranking loss. UTMOS reports utterance-level MSE of 0.165 and SRCC of 0.897 on the VoiceMOS 2022 main track (UTMOS).

For system-level comparison, the metadata-fusion model reports system SRCC of 0.934 and system MSE of 0.088 (Using Rater and System Metadata…). MBNet reports 0.949 system SRCC on VCC 2018 and 0.920 on VCC 2016 (MBNet).

We will reimplement UTMOS's single-model core rather than its full stacking ensemble. Every variant will share the same backbone, training data, and hyperparameter search budget, so that differences come from the rater model alone.

On public data without order information, our model reduces to static-plus-amortized inference. Beating UTMOS there means matching its utterance SRCC for seen raters and exceeding it for held-out raters. On our new collected data there are no published numbers, so all baselines are run fresh.

Ablation that isolates the key ingredient. The decisive ablation is the covariate-only state model. It is identical to ours, with position, previous rating, latency, and replays as inputs, but it has no stochastic latent. If our model does not beat this one on anchor test-retest prediction and on held-out-rater likelihood, then the latent-state idea adds nothing beyond feature engineering. We would report that outcome as the finding.

A second ablation swaps the ordinal likelihood for MBNet's clipped squared error. This shows whether the observation model matters independently of the state.

Why not just…?

Why not just average more annotators? Averaging only removes rater state if presentation order is fully balanced across raters. Real benchmarks are sparse. The metadata paper notes that many systems have only one or two utterances rated by few raters, which makes sample means unstable. MBNet shows that modeling individual judgments beats training on means.

Averaging is still the honest gold standard, so we make it the target rather than the rival. We collect a large-panel, fully randomized consensus. We then measure how many ratings each method needs to reach a given SRCC against it. If plain averaging reaches the consensus as fast as our state-corrected estimate, the practical case collapses.

Why not just add trial position and previous rating as input features to a static listener embedding? This is the most serious threat. Contrast and fatigue effects may be largely captured by a few hand-chosen covariates. We do not argue around it. The covariate-only model is our central ablation, and the repeated anchors give a direct test. If the residual drift on anchors after covariate correction is indistinguishable from repeat noise, the stochastic latent is unnecessary and the simpler model should be used.

Why not just z-score each rater using calibration clips? Per-rater linear normalization handles static offset and scale for new raters. It cannot represent within-session change. We include it as a baseline for the unseen-rater task: normalization from the same $k$ calibration ratings versus amortized state inference. Our model must win at small $k$ to justify itself.

Why not just ask a large audio language model to judge? Reported results suggest these models are weaker judges than specialized predictors. Gemini 3 Pro reaches 65.80% musicality accuracy against 78.20% for a specialized reward model on CMI-RewardBench. Gemini-2.5-Flash reaches 69.1% against 77.2% for SpeechJudge-GRM on SpeechJudge-Eval.

More fundamentally, such a judge predicts a population-level opinion. It does not answer whether the human target itself is drift-contaminated. Our cleaned targets would still be useful for training or evaluating such a judge.

Proposed Experiments and Decisive Checks

The first check is existence. On the collected data, do repeated anchors show within-session score change? Is that change systematic, meaning consistent in direction across raters and dependent on preceding clips, beyond test-retest noise? We will estimate this with a mixed model before any neural modeling. A null here ends the dynamic-state line, and the project reverts to static-plus-amortized inference.

The second check is prediction. On held-out raters, given $k \in \{3, 5, 10, 20\}$ calibration ratings, we compare the log-likelihood and exact-match accuracy of each subsequent rating across the nested models. We also predict a rater's later score on a repeated anchor from their earlier score plus intervening history.

The third check, the one that bears on metric divergence, is ranking reproducibility. We split raters into disjoint halves many times. For each half, we estimate system rankings from the audio term $q$ alone, and we measure SRCC between halves and against the large-panel consensus. A method that truly removes rater state should give higher split-half agreement than the averaged-score and static-embedding baselines, at equal rating budgets.

For external comparability, we report utterance and system SRCC and MSE on the VoiceMOS 2022 main and out-of-distribution tracks and on VCC 2016, against the published figures above. The out-of-distribution track has only 32 raters, which makes it a natural small-$k$ test for unseen-rater inference.

The final, more exploratory check uses SongEval's five dimensions. SongEval reports strong entanglement across coherence, musicality, and naturalness. If part of that correlation is shared rater state rather than a shared musical property, then conditioning on $s_{j,t}$ should reduce the correlation between dimensions in the residual ratings. A reduction would be one concrete mechanism behind the entanglement problem that SongEval and CMI-RewardBench both report.