When synthetic speech is sped up, as it is for blind screen-reader users, 2x podcast listeners, and video dubbing, two published findings disagree about how to do it. An older human listening study found that simply squeezing a normal-rate voice uniformly in time beat cleverer duration models at 3x speed. A recent neural study found the opposite: a duration model that stretches and shrinks individual sounds unevenly to hit a target length roughly halved errors at 2x. The two studies differ in era, rate, and system. They also differ in how intelligibility was measured. The neural study scored ordinary meaningful sentences with a speech recognizer, and that recognizer, like a human, can guess a blurred word from context. This note proposes to test whether the neural advantage survives once guessing is removed. It uses semantically unpredictable sentences: grammatical but nonsensical sentences such as "The green desk sang a heavy potato," where context gives no help. Humans and a range of automatic metrics would score the same sped-up audio.
The core idea fits in two sentences. Build a factorial listening experiment that crosses speaking rate, acceleration method, and whether the sentence is predictable or nonsensical, and score every cell with both human transcribers and automatic metrics. This tells us whether the "better fast-speech method" ranking is a property of the audio or of the evaluator's language prior, and which cheap automatic metric, if any, reproduces the human ranking at high speed.
Core Claim
The claim is that published conclusions about fast-speech methods depend on the evaluation material and the evaluator, and that this dependence can be measured directly.
The first piece of evidence is the conflicting pair of studies. Intelligibility analysis of fast synthesized speech had humans transcribe hidden-semi-Markov-model synthesis. At 3x speed, linear waveform compression by overlap-add (WSOLA) had under 20% word errors, against over 40% for non-linear duration methods. Its explanation was that uniform compression keeps "super-human" crisp consonant boundaries. Total-Duration-Aware Duration Modeling for Text-to-Speech Systems conditioned a neural duration predictor on the target total length and compressed vowels and pauses more than consonants. On LibriSpeech it cut recognizer word error rate at 2x from 15.1% to 7.4%, with a 58.7% human intelligibility preference.
The second piece of evidence is that the recognizer used as the judge leans on context. Hearing deficits of transformer-based ASR for anechoic and spatial signals showed that Whisper's gap to human listeners widens substantially on matrix sentences, which are closed-set nonsensical sentences, compared with predictable sentences. That study was run only at normal speaking rates.
Uneven compression shortens vowels and pauses, and a context-reliant judge may forgive exactly that. We therefore predict one of two outcomes. Either the neural advantage shrinks or reverses on nonsensical sentences at 2.5–3x, or it holds, which would settle the contradiction in the neural model's favour on acoustic grounds. Both outcomes are informative.
The central measured quantity is the context benefit at rate $r$:
$$\Delta_e(r) = E_e^{\text{SUS}}(r) - E_e^{\text{pred}}(r),$$
where $E_e$ is the word error rate that evaluator $e$ (a human cohort or a machine metric) produces on nonsensical (SUS) or predictable sentences. If a metric's $\Delta$ curve departs from the human curve as rate increases, the metric is substituting its prior for acoustics exactly where fast-speech methods differ.
Research Program
Stimulus design. Materials follow the SUS design of Text-to-speech intelligibility across speech rates. That study used 6–8-word grammatical templates with frequency-binned, phonetically varied words, and tested humans only, at 200–450 words per minute, on 2012-era commercial systems with undisclosed rate control. Our predictable control set is ordinary read sentences matched to the SUS in syllable count and content-word frequency distribution. Matching these properties means the only systematic difference is semantic predictability, which addresses the lexical-familiarity confound noted for SUS lists.
Acceleration conditions. All systems are built on one normal-rate neural voice, so that acoustics are held fixed. There are four conditions. The first is WSOLA applied to the synthesized waveform. The second scales every predicted duration by the same factor. The third is a total-duration-aware non-linear duration model following Eskimez et al., reimplemented if the original is unavailable. The fourth is natural recorded speech compressed with WSOLA, which serves as a human reference as in Syrdal et al. Rates are 1.0, 1.5, 2.0, 2.5, and 3.0x, which spans roughly the 200–450+ word-per-minute range of the prior human studies.
Human listeners. Two cohorts transcribe in open response, scored by normalized edit distance. The first is sighted normal-hearing listeners. The second is a smaller cohort of experienced screen-reader users. Blind listeners were shown to cope at rates where sighted listeners fail by Synthesis of fast speech with interpolation of adapted HSMMs and its evaluation by blind and sighted listeners. Because SUS transcription is effortful (Using Pupillometry to Measure the Cognitive Load of Synthetic Speech), sessions are short and counterbalanced so that no listener hears the same sentence twice. Headphones are required, since Syrdal et al. report 6.2% fewer errors with headphones than with speakers.
Machine evaluators. The same audio is scored by four families of evaluators.
The first family is Whisper at several model sizes with standard decoding. This is the evaluator most papers use.
The second family is context-free machine listeners. One is Whisper with decoding constrained to the known SUS vocabulary, mirroring the closed-set matrix format of Hoffner et al. The other is a phoneme recognizer with no word language model, scored by phoneme error.
The third family is intrusive metrics: short-time objective intelligibility (STOI), its extended variant (ESTOI), and ViSQOL v3. Each is computed against the uncompressed-rate reference in two ways: after linear compression of the reference, and after dynamic-time-warping alignment. ViSQOL v3's authors already document that such metrics penalize benign timing shifts. We test whether warping repairs that penalty for non-linear compression.
The fourth family is non-intrusive neural predictors, such as STOI-Net and the Whisper-embedding model iMTI-Net from Speech Intelligibility Assessment with Uncertainty-Aware Whisper Embeddings and sLSTM. Both were trained only on normal-rate noisy speech.
Analysis. Each evaluator is judged on three things. The first is whether it reproduces the human ranking of acceleration methods within each rate, measured by Kendall's tau over the four conditions. The second is its correlation with human error across all cells. The third is how far its context-benefit curve $\Delta_e(r)$ departs from the human curve. The non-linear intelligibility plateau reported by Study on the Correlation between Objective Evaluations and Subjective Speech Quality and Intelligibility means that rank-based measures, not linear fits, are the primary statistics.
Baselines
Naive evaluation reference. The do-nothing reference is the current default: Whisper word error rate on accelerated predictable sentences, as in Eskimez et al. The strongest reported number for the method comparison it supports is the drop from 15.1% to 7.4% word error rate at 2x for total-duration-aware modeling over its baseline, from Total-Duration-Aware Duration Modeling for Text-to-Speech Systems. We rerun this exact protocol on our audio, so that it and every alternative evaluator see identical waveforms. Any evaluator we propose beats it only if it matches the human SUS ranking of methods at 2.5–3x where this default does not.
Strongest acceleration competitor. On the synthesis side, the strongest competitor to the neural method is WSOLA on a normal-rate voice. It is the method that won at 3x in human tests, with under 20% versus over 40% errors in Intelligibility analysis of fast synthesized speech. Both it and the total-duration-aware model use the same voice, text, and target durations, and neither gets any rate-specific tuning. We use no pre-committed margin. Beating WSOLA means a non-overlapping bootstrap interval on human SUS error at matched rate.
Strongest learned metric. The strongest existing learned intelligibility predictor is iMTI-Net. It reports a linear correlation of 0.7817 with human intelligibility on TMHINT-QI(S), where TMHINT-QI(S) is a Mandarin corpus of noisy and speech-enhanced utterances with human intelligibility ratings. For fast SUS speech we found no reported result on Lacuna. We run iMTI-Net off the shelf, and also after fine-tuning on human labels with leave-one-rate-out splits, giving the same label budget available to any calibrated metric we build.
Key ablation. The ablation that isolates the proposal's ingredient is running every evaluator on predictable versus SUS material from the same acceleration conditions. This separates the effect of removing context from the effect of rate.
Why not just…?
Why not just run Whisper on fast SUS and skip the human panel? This is cheap, and it may be most of what practitioners need. However, Hoffner et al. show Whisper's gap to humans on nonsensical sentences reaches an average of 7.9 dB in cafeteria and 8.6 dB in office reverberation. Its errors on context-free material therefore do not track humans even at normal rate. We found no evidence either way at fast rates. Without human labels, a Whisper result on SUS cannot be called intelligibility. The panel is the calibration that makes every later automatic run trustworthy.
Why not just use a phoneme recognizer with no language model? This is a real threat to the need for anything more elaborate. It removes the prior by construction and may track humans well. We therefore include it as a first-class candidate rather than a straw man. If its $\Delta$ curve and method rankings match the human ones, the practical conclusion is simply "use it," and this study is what licenses that recommendation. The risk is that humans also use syntax and word knowledge on SUS, so a purely phonetic listener could be too harsh. The human-versus-phoneme comparison on SUS measures this directly.
Why not just stay with predictable sentences, since that is what people actually listen to? Realism argues for predictable material, and we keep it as one arm. But Syrdal et al. and the Blizzard 2005 evaluation (The blizzard challenge - 2005) both motivate SUS precisely because context hides segmental errors. Whether hidden errors change which acceleration method wins is the question under test. If $\Delta$ turns out flat across methods and rates, we will report that predictable sentences suffice, which would be a useful negative result.
Why not just fine-tune a neural intelligibility predictor on fast speech? Fine-tuning needs exactly the human labels this study collects. The leave-one-rate-out experiment tests whether a predictor fine-tuned this way extrapolates to unseen rates. If it does, fine-tuning is the right deployment route, and the dataset produced here is the resource that enables it.
Proposed Experiments and Decisive Checks
The first check is the ranking-flip test. At 2.5x and 3x on SUS, does the sighted cohort rank the total-duration-aware model above WSOLA, as the recognizer on LibriSpeech would suggest, or below it, as Valentini-Botinhao et al. found with humans? A reversal between the predictable and SUS arms would show that the published neural advantage depends on context.
The second check is metric agreement. For each evaluator, we compute its Kendall tau with the human method ranking at each rate. We expect intrusive metrics with linear-compressed references to fail specifically on the non-linear condition. We expect warping to repair part of that failure.
The third check is context-benefit divergence. We test whether standard Whisper's $\Delta(r)$ grows with rate faster than the human $\Delta(r)$. If so, that is direct evidence that recognizer scores lean increasingly on the prior as speech speeds up.
The fourth check is a cohort split. We repeat the rankings for screen-reader users and ask which cohort each metric tracks. This matters because the people who most need fast speech are the ones least like the listeners the metrics were calibrated on.
Success means a clear answer to the first check, and at least one automatic evaluator that reproduces human rankings at every rate within bootstrap noise. We will release the stimuli and transcriptions so the benchmark can be reused.
References
- Intelligibility analysis of fast synthesized speech
- Total-Duration-Aware Duration Modeling for Text-to-Speech Systems
- Hearing deficits of transformer-based ASR for anechoic and spatial signals
- Text-to-speech intelligibility across speech rates
- Synthesis of fast speech with interpolation of adapted HSMMs and its evaluation by blind and sighted listeners
- Using Pupillometry to Measure the Cognitive Load of Synthetic Speech
- ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metric
- STOI-Net: A Deep Learning based Non-Intrusive Speech Intelligibility Assessment Model
- Speech Intelligibility Assessment with Uncertainty-Aware Whisper Embeddings and sLSTM
- Study on the Correlation Between Objective Evaluations and Subjective Speech Quality and Intelligibility
- The blizzard challenge - 2005: evaluating corpus-based speech synthesis on common datasets