Models that predict how well a hearing-impaired person will understand a sentence in noise are scored by the Pearson correlation between their predictions and the percentage of words the listener actually got right. Progress on the Clarity Prediction Challenges has slowed. The best systems sit near a correlation of 0.79 and an error of about 23 percentage points, and nobody knows how much of the remaining error is the model's fault. Some of it cannot be the model's fault. Each human score comes from a single short sentence, so it is a coin-flip-like count over a few words and is noisy in itself. Some of the rest comes from the listener: people with the same audiogram (the standard hearing test of quiet-tone thresholds) understand speech in noise very differently. No acoustic model can see that difference unless it is given information about the listener.
We propose to fit an explicit statistical measurement model to the challenge data. Structural equation modeling is the name for a family of such models, which treat observed scores as noisy readings of hidden quantities. Ours will split each word-correct score into four parts: sentence-level counting noise, a hidden "listener ability" factor, a hidden "scene degradation" factor, and their interaction. From this split we compute the largest correlation any predictor could reach, both for listeners seen in training and for new listeners. We also explain a puzzling plateau in few-shot personalization, and we measure how much of listener ability the audiogram actually captures. The output is a calibrated ceiling and an error budget for the benchmark. It is not another predictor.
Core Claim
The headline numbers on the Clarity benchmarks mix three different things: model error, unreliability of the human target, and listener variance that the inputs cannot reveal. Separating them changes how the existing results should be read. Our first claim is that a substantial share of the gap between current systems and a perfect correlation is irreducible counting noise. Our second claim is that the remaining open-set (unseen-listener) gap is set mainly by how much listener variance the audiogram fails to measure. Our third claim is that the plateau after about 16 support samples, reported for few-shot personalization by No Audiogram, follows from a simple ratio of listener variance to per-response noise. If so, it is a property of the data rather than a weakness of the network.
Two relations carry the argument. For a sentence with $n$ scored words and true word-recognition probability $p$, the observed proportion has sampling variance $p(1-p)/n$. Write $\rho$ for the reliability of the observed score, meaning the share of its variance that is true variance rather than counting noise. Then the classical attenuation result bounds the correlation of any predictor with the observed score:
$$r_{\max} = \sqrt{\rho}, \qquad \rho = \frac{\mathrm{Var}(p)}{\mathrm{Var}(p) + \mathbb{E}\left[p(1-p)/n\right]}.$$
The second relation concerns how well a listener's ability can be estimated from data. Let $\tau^2$ be the variance of the latent listener factor and $\sigma^2$ the residual per-response variance on the model's scale. If the listener factor is estimated from $k$ scored responses, the reliability of that estimate is
$$\rho_k = \frac{\tau^2}{\tau^2 + \sigma^2/k}.$$
This curve predicts where support-sample personalization should saturate. If the fitted $\tau^2/\sigma^2$ places most of the gain below $k \approx 16$, the plateau seen in No Audiogram is explained.
Research Program
The data are the CPC2 set, which contains over 24,000 binaural hearing-aid-processed samples with listener audiograms and word-correct percentages. For replication we also use CPC 2022 and CPC1. The model is a binomial latent-variable model on the logit scale. The logit scale removes the ceiling and floor distortions that deflate plain Pearson correlations, which is one of the known limitations of correlation analysis.
The model has three parts. The first is a scene factor, measured by several objective indicators of the same processed signal: HASPI, better-ear STOI, the glimpse-proportion metrics of Tang and Cooke (which correlate 0.81–0.92 with human scores for normal-hearing listeners), and the prediction of a frozen foundation-model predictor. The second is a listener factor, measured by the audiogram's frequency bands and by the listener's held-out word scores. The third is an optional listener-by-scene term, either a listener-specific slope on the scene factor or a listener-by-system effect.
This design borrows its measurement logic from Aligning MLLM Benchmark With Human Preferences via Structural Equation Modeling. That paper used factor loadings, redundancy (variance inflation) and discriminant-validity (heterotrait–monotrait) diagnostics to prune vision-language benchmark tasks, and it raised correlation with human preference to 0.7359. We reuse those diagnostics for a narrower purpose. We ask whether the zoo of intelligibility metrics measures one scene factor or several, and whether the audiogram is a noisy indicator of the same listener ability that held-out scores reveal or a partly different construct.
The audiogram question sharpens a finding from ELO-SPHERES. ELO-SPHERES reported that audiograms explain about 40% of intelligibility variation and that one-hot listener identity was far more informative, but it could not say what that identity encodes. A loading-based decomposition says how much of the listener factor the audiogram carries. It also says whether the rest behaves like a single offset, a general speech-in-noise ability, or like listener-specific sensitivity to particular processing.
The parts are fitted in three stages. First, from per-sentence word counts recovered from the prompts, we compute the counting-noise ceiling analytically. Second, we fit the latent model with and without each indicator block. Third, we derive two quantities. One is the ceiling for seen listeners, where the listener factor is known. The other is the ceiling for open-set listeners, where only the audiogram is available. We validate everything on simulated data with known parameters before trusting it on real data.
Baselines
The naive reference point is the better-ear STOI or HASPI baseline treated as the predictor. ELO-SPHERES reports an error of 27.4% for better-ear STOI on CPC 2022, and Speech Foundation Models on Intelligibility Prediction for Hearing-Impaired Listeners reports a HASPI error of 28.70 on CPC2. Our ceilings must place these baselines far below the bound. If they did not, the decomposition would be wrong.
The strongest closed-set competitor is SSIPNet from No Audiogram. It conditions a foundation-model predictor on a listener's previous scores and reports an average RMSE of 23.430 and an average normalized correlation of 0.811 on CPC2, against 26.160 and 0.794 for its baseline. We rerun its released configuration where possible. Otherwise we take its published numbers on the identical CPC2 splits. We compare its correlation with our seen-listener ceiling and its support-sample curve with our predicted $\rho_k$ curve.
The strongest audiogram-only competitor is the foundation-model ensemble of Cuervo et al., which reached an RMSE of 23.86 on CPC2. Its correlation is compared against our open-set ceiling.
The comparison stays fair because our model never sees test labels for ceiling estimation. Ceilings are computed from training folds and then evaluated on held-out folds. All predictors are scored on identical items.
This project does not "beat" these systems by predicting better. It succeeds if two things hold. The ceilings must be consistent with them, with no system exceeding its ceiling on held-out data. And the ceilings must be informative, separating the remaining error into counting noise, unmeasured listener variance and true model error with tight intervals.
The key ablation compares two forms of the listener part. One is a random listener intercept with no indicators, which is just a mixed-effects logistic regression. The other is the multi-indicator listener factor. This isolates what the structural-equation measurement model adds over standard random effects.
Why not just…?
Why not just compute a split-half or binomial noise ceiling? For the counting-noise part, we do exactly that. It is analytic and needs no latent model. The simple ceiling, however, treats all non-counting variance as predictable. In the open-set track, much of it is listener variance that the inputs do not identify, so the simple ceiling overstates what open-set systems can reach. The experiment that settles this asks whether the top open-set correlations, 0.76 on CPC1 and 0.75 for MBI-Net, sit much closer to our audiogram-only ceiling than to the naive one.
Why not just fit a mixed-effects model with a random listener intercept? This is the real threat, because a one-factor model with only an intercept is a mixed model. We adopt it as the core estimator rather than argue around it. The structural-equation layer earns its place only if letting audiogram bands and held-out scores load on a shared factor tells us something the intercept cannot. One such result would be a large unique component outside the audiogram. Another would be evidence that listener effects interact with processing system, which would show up as better held-out fit for listener-specific slopes. If the multi-indicator model and the intercept model give the same ceilings and the same fit, we will report that the simpler mixed model suffices.
Why not just train a better listener-conditioned network, as SSIPNet or ELO-SPHERES do? Better prediction cannot say how much better prediction is possible. A network approaching our ceiling is the expected outcome, not a counterexample. Our model makes a falsifiable claim the networks do not: the saturation point of support-sample learning, computed from $\tau^2/\sigma^2$ before looking at SSIPNet's curve.
Proposed Experiments
The first decisive check is recovery. On simulated CPC2-shaped data with known listener variance, scene effects and word counts, the fitted ceilings must match the true values within their intervals.
The second check is the plateau. The $\rho_k$ curve, fitted on training listeners, must place saturation near the 16-sample plateau reported by No Audiogram. If saturation is predicted far earlier or later, the plateau reflects the network rather than the data, and that is equally worth reporting.
The third check is the ceiling ordering. On held-out folds of CPC2, CPC 2022 and CPC1, every published system must fall below its track's ceiling. The HASPI and STOI baselines must sit far below. The gap between top systems and the ceiling becomes the honest estimate of remaining modelling headroom.
The fourth check is the audiogram's share. The audiogram's share of listener-factor variance should come out near the roughly 40% that ELO-SPHERES reports when the counting noise is ignored. After correction for counting noise, we expect the share to change, and we report its direction.
The fifth check is a metric-redundancy analysis. Variance inflation and discriminant validity among HASPI, STOI, glimpse metrics and foundation-model predictions show whether these metrics measure one scene construct. This tells future work which inputs are worth combining.
References
- No Audiogram: Leveraging Existing Scores for Personalized Speech Intelligibility Prediction
- Glimpse-Based Metrics for Predicting Speech Intelligibility in Additive Noise Conditions
- Aligning MLLM Benchmark With Human Preferences via Structural Equation Modeling
- ELO-SPHERES intelligibility prediction model for the Clarity Prediction Challenge 2022
- Speech Foundation Models on Intelligibility Prediction for Hearing-Impaired Listeners
- MBI-Net: A Non-Intrusive Multi-Branched Speech Intelligibility Prediction Model for Hearing Aids