Automatic speech quality predictors learn to imitate human listening tests. They are trained on the mean opinion score (MOS), the average of many listeners' 1–5 ratings of a clip. The best of them, such as UTMOS and MBNet, are trained with every rating and every listener identity pooled on one server. Real quality feedback increasingly arrives from people's own devices: a thumbs-up on a call or a rating of a synthetic voice. That audio and those listener profiles are exactly what one should not centralize. Federated learning trains a shared model by sending weight updates from devices instead of raw data, so it looks like the natural fix. The difficulty is not only privacy. It is psychometric. Each device's ratings come from one listener, on that listener's own scale, and that scale is stretched to fit whatever range of audio the listener happened to hear. Cooper et al. documented this "range-equalizing bias": someone who only hears good audio uses the whole scale on small flaws. As a result, a 3 on one phone and a 3 on another are not the same number, and averaging models trained on them mixes incompatible targets.
We propose that each device keep a small private calibration: the thresholds that convert a shared latent quality into that listener's personal 1–5 categories. The server aggregates only the shared quality model. To tie all the private scales together, every device also rates a handful of public anchor clips spanning the quality range. This is the idea behind the hidden reference and hidden anchor in MUSHRA listening tests (the ITU-R BS.1534 protocol), moved into the federated setting. The central claim is that anchored per-listener calibration recovers most of the gap between naive federated averaging and centralized, listener-aware training. It should also clearly beat the cheap fixes: per-client score normalization and within-client ranking losses. Both of those discard between-listener information whenever listeners heard different ranges of audio.
What exists and what is missing
The research file contains no federated MOS predictor. Federated speech work targets generation or representation learning. FedSpeech trains multi-speaker text-to-speech across devices with pruning masks and private speaker modules, and still evaluates with a centralized MOS test. Federated Self-supervised Speech Representations pretrains wav2vec 2.0 federatedly and evaluates only speech recognition. These papers supply methods and constraints for us. They do not address quality assessment.
On the quality side, three centralized papers motivate the design by showing that listener identity carries real signal. MBNet splits prediction into a clip mean and a per-judge bias, using judge IDs as a lookup table. Using Rater and System Metadata shows that rater and system metadata explain much of BVCC's variance. UTMOS adds learned listener embeddings. All three assume the server can see every listener ID and every rating. Our proposal keeps their key ingredient, a listener-specific correction, but makes it private and structured as an ordinal link rather than an additive bias. That structure lets it absorb range-dependent stretching and not just an offset.
MOSPC supplies a second method: pairwise ranking losses. A ranking loss computed within one listener's ratings is naturally immune to that listener's monotone scale distortion. It is also blind to how that listener's whole range sits relative to others, which is the failure mode the anchors are meant to fix. DDOS argues for predicting the distribution of opinion scores rather than the mean. An ordinal model with per-listener thresholds yields such a distribution by construction.
Core model
Let $f_w(x)$ be a shared network with parameters $w$ that maps a clip $x$ to a scalar latent quality. Listener $k$ holds private, ordered thresholds $\theta_{k,1} < \dots < \theta_{k,4}$. The probability that listener $k$ rates clip $x$ at category $c$ or below is modeled as
$$P(r \le c \mid x, k) = \sigma\big(\theta_{k,c} - f_w(x)\big),$$
where $\sigma$ is the logistic function. This is a cumulative-link ordinal model. A listener who compresses or stretches their scale simply has differently spaced thresholds.
On device, client $k$ minimizes the ordinal negative log-likelihood over its private ratings $\mathcal{D}_k$ and its ratings of a public anchor set $\mathcal{A}$ of $m$ clips (for example $m=10$, from clean reference to severely degraded). It adds a proximal term in the style of FedProx:
$$\mathcal{L}_k(w,\theta_k) = -\sum_{(x,r)\in \mathcal{D}_k \cup \mathcal{A}_k} \log P(r \mid x, k) + \frac{\mu}{2}\|w - w_t\|^2.$$
Here $w_t$ is the current global model and $\mu$ is the proximal weight. The thresholds $\theta_k$ never leave the device. Only updates to $w$ are aggregated. Because every client's anchor ratings pass through the same $f_w$ on the same clips, the anchors pin all private scales to one latent axis. Without them, the joint model is identifiable only up to per-client shifts whenever client audio ranges do not overlap. Anchor audio is public, so rating it reveals only the listener's taste on known clips.
To respect the edge memory limits reported in Federated Self-supervised Speech Representations, where wav2vec 2.0 Large runs out of memory on 8 GB devices, $f_w$ is a frozen, small self-supervised backbone with a trainable lightweight head (a few-layer attention pooler plus a small adapter). A NISQA-style CNN–self-attention model (NISQA) is a second, fully trainable variant. Every compared method uses the same backbone and head.
Research Program
The work has three parts.
The first is a federated benchmark for MOS prediction, of which the file contains none. We will use BVCC, where each clip has eight listener-level ratings with listener identities, and VCC 2018, with 267 judges and four per clip. We treat each listener as a client holding only their own ratings. This partition inherits the real range effects of the original listening sessions. A controlled variant then re-selects each client's clips into narrow quality bands and applies a range-equalizing rescaling of that client's ratings. The rescaling is fitted to the direction and rough size of the effect Cooper et al. report, which lets us dial the severity of scale mismatch. The public anchor ratings for simulated clients come from held-out BVCC clips that each client's listener actually rated where possible. Otherwise they are generated through the same simulated transform, and we report both cases separately.
The second part is the method and its ablations. We will test anchors on and off, ordinal thresholds versus an additive MBNet-style bias, and the number of anchors, including the smallest $m$ that suffices.
The third part is a small real listening study that tests whether the simulated range effect matches reality. Listeners are split into groups that hear only high-, only low-, or mixed-quality BVCC-style stimuli, and every group also rates the same anchor set, mirroring Cooper et al.'s design. We then run the federated protocol on those ratings as if each listener were a device.
Baselines
The naive reference point is FedAvg with mean-squared-error regression on raw individual ratings. Each client trains the shared head on its own 1–5 scores and the server averages weights in proportion to client data size. It uses the same backbone, the same number of communication rounds, and the same per-round local compute as our method. MBNet's ablation, where removing the bias branch dropped VCC 2018 system-level SRCC from 0.949 to 0.934, suggests this baseline will be hurt by rater bias. We have no reported result on Lacuna for any federated MOS predictor, so this number must be produced here.
The strongest federated competitor is FedProx on the same regression loss, with per-client z-scoring of ratings before training. Flower Hub found FedProx best or second-best across its heterogeneous tasks, and z-scoring is the obvious calibration fix. We tune its proximal weight and learning rate with the same grid budget as ours.
The second strong federated competitor is a federated MOSPC-style within-client pairwise ranking loss, which is invariant to each listener's monotone scale distortion.
The centralized reference is the method a privacy-free system would use. We run UTMOS-style listener-embedding training and MBNet on pooled data with the same small backbone, so the comparison isolates federation rather than model size. For scale context, the best published systems report the following. UTMOS reports utterance-level SRCC 0.897 and MSE 0.165 on the BVCC main track. The metadata model of Chinen et al. reports utterance-level SRCC 0.877 and system-level SRCC 0.934. MBNet reports system-level SRCC 0.949 on VCC 2018. These use much larger backbones, so they are a ceiling, not a target.
The key ablation is our method without anchors. It isolates whether the public calibration set, rather than the ordinal parameterization alone, does the work.
Our method counts as beating a competitor if it improves utterance-level SRCC and reduces MSE against held-out clip means on the BVCC test set. The improvement must hold across three seeds with non-overlapping confidence intervals. The main success criterion is closing at least half of the gap between the best federated baseline and the same-backbone centralized listener-aware model. The gains should grow with the severity of the simulated range mismatch.
Why not just…?
Why not just z-score each client's ratings and run FedProx? Z-scoring removes each listener's offset and spread, but it also removes real differences in the audio they heard. A listener who only heard poor voice conversions ends up with the same mean as one who only heard studio recordings. It is exactly correct only when all clients heard the same quality distribution. It becomes wrong precisely under range-equalizing bias, where the observed spread reflects both the stimuli and the listener. The controlled range-severity sweep will test this directly. If z-scoring matches our method at every severity, the ordinal-plus-anchors machinery is unnecessary and we will say so. This is the most serious threat.
Why not just use a within-client ranking loss, since rankings ignore scale? Within-listener rankings are indeed robust. MOSPC shows that pairwise training improves ranking, reporting VCC 2018 system-level KTAU 0.938. But a client that only heard high-quality clips can teach the model orderings only within that band. Nothing ties that band to the bands other clients heard. With little clip overlap across clients, the global scale is unidentified. The narrow-band partition is designed to expose this, and our anchor ablation shows whether anchors are what fixes it.
Why not just pool anonymized ratings on a server and train MBNet? Where pooling is allowed, one should, and the centralized reference measures what federation costs. The premise here is device-originated feedback about private audio. In that setting, even anonymized per-listener rating histories can re-identify people, and FedSpeech notes that speech-derived signals invite inference attacks. We do not claim federation is better than pooling. We claim that when pooling is disallowed, the loss can be small.
Why not just freeze features and federate a ridge regressor? It is cheap and fits within the memory limits. UTMOS found such weak learners useful only as ensemble members and much weaker alone. More importantly, a ridge regressor on raw ratings inherits the same scale mismatch. We include a federated linear-head variant, with and without anchored ordinal calibration, to show whether the benefit comes from calibration or from head capacity.
Proposed Experiments
The main experiment trains all methods on the listener-as-client partitions of BVCC and VCC 2018. We report utterance-level and system-level SRCC, LCC, and MSE against held-out clip means. We also report the Kendall tau between each client's calibrated predictions and their actual held-out ratings, which measures how well the model personalizes.
The out-of-domain check trains on BVCC and tests zero-shot on the Chinese BC2019 set. The VoiceMOS Challenge found cross-lingual transfer fragile, and we ask whether federation with anchors degrades that further. A known failure point here is that the anchor clips are English. If transfer collapses relative to the centralized reference, anchors may need to be refreshed per language.
The decisive checks are three. First, the gap between our method and z-scored FedProx should widen monotonically as the simulated range mismatch grows. It should also appear in the real three-group listening study, not just in simulation. Second, removing the anchors should hurt most in the narrow-band partition. Third, performance as a function of anchor-set size should saturate at a small $m$, which is what makes the protocol practical to deploy.
Communication volume and on-device peak memory are reported for each method, so that the edge feasibility claim is measured rather than assumed. We state plainly that shared weight updates can still leak information, and that secure aggregation and differential privacy are left for future work.
References
- UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022
- MBNet: MOS Prediction for Synthesized Speech with Mean-Bias Network
- Investigating Range-Equalizing Bias in Mean Opinion Score Ratings of Synthesized Speech
- FedSpeech: Federated Text-to-Speech with Continual Learning
- Federated Self-supervised Speech Representations: Are We There Yet?
- Using Rater and System Metadata to Explain Variance in the VoiceMOS Challenge 2022 Dataset
- MOSPC: MOS Prediction Based on Pairwise Comparison
- DDOS: A MOS Prediction Framework utilizing Domain Adaptive Pre-training and Distribution of Opinion Scores
- NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets
- Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deployment