Speech recognition systems can now listen to a child read aloud and produce the numbers teachers use to judge reading fluency: words correct per minute (WCPM), speaking rate, and the number and length of pauses. The best published evidence that these automatic numbers are trustworthy comes from correlations with human-scored versions of the same recordings. A recent study reports Pearson $r \ge 0.70$ for 12 of 15 fluency measures. That result, however, was computed on a single pool that merged Dutch and Flemish children from three corpora. Clinical prediction research has learned that this kind of evaluation misleads. A model can look excellent on the data it was built on and then degrade at a new hospital. The drop usually shows up less in ranking patients than in absolute calibration, meaning the predicted values come out systematically too high or too low. For reading assessment, a calibration failure matters directly, because schools act on fixed cut-offs: a child whose automatic WCPM falls below a threshold gets flagged for intervention.
We propose to run, for the first time, a clinical-style external validation of an automatic reading-fluency pipeline. All components are built and fixed on Dutch children (JASMIN-NL). They are then applied unchanged to Flemish children (JASMIN-VL) and to separately collected Dutch-language cohorts (SERDA-NL, Chorec). We ask three questions for each measure. Does its ranking of children survive the transfer? Does its absolute scale survive? Do at-risk flags made at a fixed threshold still agree with the flags human scoring would produce? We then ask how few locally hand-transcribed recordings a new site needs to repair whatever breaks.
Core Claim
Our claim is that correlation-only validation of automatic fluency measures hides two separate failures under transfer. The first is loss of discrimination: the automatic measure no longer ranks children the way human scoring does. The second is loss of calibration: the ranking survives, but the automatic measure is shifted or stretched relative to human scoring. We expect the second failure to dominate for timestamp-derived pause measures. Their values depend on voice-activity thresholds and recording conditions that differ between sites. We expect WCPM and speaking rate to be comparatively robust.
If this holds, two practical consequences follow. First, the pooled $r \ge 0.70$ result is compatible with substantial misclassification of at-risk children at a new site. Second, a cheap local step, a linear rescaling fitted on a handful of hand-transcribed local recordings, recovers most of the lost decision accuracy without retraining the recognizer. This second point is the concrete version of the "all models are local" argument, applied to reading assessment.
Positioning Against Prior Work
The fluency pipeline we validate comes from Can ASR generate valid measures of child reading fluency?. That paper runs Whisper Large-v2 with word timestamps, disfluency detection and voice-activity detection, then aligns the transcript to the prompt text to derive 15 pacing, smoothness and phrasing measures. It pooled JASMIN-NL, JASMIN-VL and SERDA-NL into 244 recordings from 131 children. We reuse its measure definitions and its best configuration (whispert_vad_dis). We differ by splitting where it pooled, by fixing every parameter before external testing, and by reporting calibration and decision agreement rather than correlation alone.
Two other lines of work motivate the proposal by showing strong numbers that were never tested outside their own cohort. Automated Estimation of Oral Reading Fluency During Summer Camp e-Book Reading with MyTurnToRead reports speaker-level $r = 0.93$ between automatic and human WCPM despite a median 40% word error rate, but aggregates across camps. The Tball work, Estimation of children's reading ability by fusion of automatic pronunciation verification and fluency detection and A Bayesian network classifier for word-level reading assessment, reaches correlations of 0.91 and 0.921 under internal cross-validation in one Los Angeles population. The second paper also notes degradation for older children, which a pooled evaluation hides. Automatic assessment of children's reading level tunes its scoring and bias compensation within the same 80-child Chorec cohort it evaluates on.
One result constrains our design. Automatic Assessment of Oral Reading Accuracy for Reading Diagnostics reaches MCC 0.63 on word-error diagnosis only when the language model includes manual transcriptions of the test children's own errors. With prompts alone, agreement falls to 0.28. Test-cohort transcripts are exactly what external validation forbids. We therefore restrict the locked pipeline to prompt-only information, and we treat word-level error diagnosis as a secondary, expected-to-be-weak measure rather than a headline.
The validation methodology comes from clinical prediction. Prediction of Outcome in Acute Lower Gastrointestinal Bleeding Using Gradient Boosting supplies the protocol: derive at one hospital, fix everything, test at another. Developing and validating COVID-19 adverse outcome risk prediction models from a bi-national European cohort of 5594 patients shows the size of drop that transfer can produce: mortality AUC of about 0.90 internally versus 0.74 on UK Biobank. All models are local: time to replace external validation with recurrent local validation documents the Epic Sepsis Model falling from an advertised 0.73–0.83 AUC to 0.60–0.64 externally. It argues for site-level recalibration and monitoring, which motivates our local-repair experiment. Radiomics in prostate cancer imaging for a personalized treatment approach - current aspects of methodology and a systematic review on validated studies identifies acquisition-hardware variation as the main source of instability in handcrafted features. This is the analogue of our hypothesis about pause measures and recording conditions.
Research Program
The pipeline has three stages, all fixed on JASMIN-NL before any external data are touched. The first stage is the Whisper configuration, including its voice-activity and disfluency settings. The second stage computes the fluency measures: the prompt aligner, the pause-length floor, and the rule for classifying pauses as occurring within or between sentences.
The third stage is a per-measure calibration map from automatic to human scale,
$$\hat{y} = a + b\,x,$$
where $x$ is the automatic value of a measure for one recording, $\hat{y}$ is the predicted human-scored value, and $a, b$ are fitted by least squares on JASMIN-NL. Fixed at-risk thresholds are also set here. For each measure we use the 25th percentile of human-scored values within coarse age bands of the derivation cohort. A recording is flagged if $\hat{y}$ falls below the threshold for its age band.
On each external cohort we report three things per measure, all with child-level bootstrap intervals because children contribute several recordings. The first is discrimination: the Spearman correlation between automatic and human values, computed within the cohort. The second is calibration, borrowed from clinical practice: the mean offset between $\hat{y}$ and human $y$, and the slope of $y$ regressed on $\hat{y}$. Perfect transfer gives offset 0 and slope 1. The third is decision agreement: Cohen's kappa and sensitivity between automatic and human at-risk flags at the fixed thresholds.
For local repair, we refit only $a$ and $b$ on $k$ randomly drawn children from the external site, with $k \in \{2, 5, 10, 20\}$, and re-measure decision agreement on the remaining children. The error in a refitted intercept should shrink roughly as $\sigma/\sqrt{k}$, where $\sigma$ is the residual spread. This is the usual standard-error argument for a mean offset, and it predicts that most of the gain arrives by about 10 children.
A secondary analysis asks whether label-free signals can warn a new site in advance that a measure will miscalibrate. The signals are the external cohort's distribution of recognizer confidence, estimated signal-to-noise ratio, and raw pause-length histograms, all of which can be computed without any human transcription. This addresses the file's noted lack of any metric for similarity between derivation and deployment cohorts.
Baselines
The naive reference point is the evaluation the field currently uses: the same pipeline, with the calibration map and thresholds fitted on the pooled JASMIN-NL, JASMIN-VL and SERDA-NL data and scored on that same pool. This reproduces the published setting. Reported numbers are Pearson $r \ge 0.70$ for 12 of 15 measures and F1 0.82 for whispert_vad_dis (Can ASR generate valid measures of child reading fluency?). No per-cohort breakdown was reported, and no reported result was found on Lacuna for fluency measures under a locked cross-cohort split. The pooled number is what practitioners currently rely on. Our claim is falsified if the locked external estimates of kappa and calibration match the pooled estimates within bootstrap intervals.
The strongest existing method is whispert_vad_dis, applied with every component fixed. It is the best-performing published configuration on these exact corpora, so any degradation cannot be blamed on a weak recognizer. As a second recognizer family, we run a Kaldi system whose language model is built from reading prompts only, in the spirit of Kaldi-PR from Automatic Assessment of Oral Reading Accuracy for Reading Diagnostics. That paper reports MCC 0.28 for error diagnosis, versus 0.63 with test-set error transcripts that we disallow. Both recognizers see identical audio, prompts and derivation data, and each receives the same small tuning budget on JASMIN-NL only.
The ablation that isolates our key ingredient separates recalibration from recognizer adaptation. We compare the frozen pipeline alone, the frozen pipeline with $k$-child linear recalibration, and the pipeline with its recognizer fine-tuned on the same $k$ children's audio and transcripts. The same human-labeling budget goes into each arm. We also include an oracle in which the calibration map is fitted on the full external cohort, which bounds what any local repair can achieve. The proposal counts as successful if $k = 10$ recalibration recovers at least most of the gap between frozen-pipeline kappa and oracle kappa, and matches or beats the fine-tuned recognizer at equal labeling cost.
Why not just…?
Why not just keep reporting pooled correlations, as the field does? Correlation is blind to offset and scale. A pause-duration measure that is uniformly 150 ms too long at a Flemish site keeps its $r$ and still moves many children across a fixed threshold. The clinical record shows that aggregate internal performance hides large transfer drops, from 0.906 to 0.74 in Developing and validating COVID-19 adverse outcome risk prediction models from a bi-national European cohort of 5594 patients. Whether this happens here is an empirical question, and the naive-versus-locked comparison answers it directly. If the two agree, that is itself a useful negative result for the field.
Why not just fine-tune the recognizer on child speech from each new site? Fine-tuning needs transcribed local audio and a retraining cycle, and it may fix recognition errors without fixing threshold-setting conditions such as recording level and room noise. This is a real competitor, so the ablation gives it the same labeling budget as recalibration. If fine-tuning wins at equal cost, we will report that recognition error, not scale shift, is the dominant failure. We would then change the recommendation accordingly.
Why not just use WCPM alone, which may transfer fine? This is the most serious threat. WCPM depends mainly on word counts and total duration, which is exactly why we expect it to be robust. If WCPM alone yields at-risk flags that agree with human flags as well as a composite of measures does, then the fragile pause measures add risk without benefit. We test this by comparing flag agreement and external-cohort kappa for WCPM alone against a composite built on JASMIN-NL. The prosody literature, for example Two methods for assessing oral reading prosody, argues that timing measures beyond rate carry comprehension information. However, that evidence comes from a single cohort, and we do not presume it transfers.
Proposed Experiments
The primary experiment derives everything on JASMIN-NL (142 recordings) and tests locked on JASMIN-VL (84 recordings). This split isolates a regional-variety shift within one corpus design. For each of the 15 measures we report within-cohort Spearman correlation, calibration offset and slope, and kappa for at-risk flags, each with child-clustered bootstrap intervals, alongside the pooled-evaluation numbers for the same measures. The decisive check is whether pause measures lose calibration, with offset intervals excluding zero or slope intervals excluding one, while keeping their rank correlations. We also check whether WCPM and speaking rate stay calibrated.
The second experiment repeats the locked test on SERDA-NL and on the story-reading portion of Chorec (80 second graders). These add a change in collection protocol on top of the regional-variety shift. SERDA-NL has only 18 recordings, so we treat it as a directional probe, reporting offsets rather than kappa. Chorec requires human fluency annotations derived from its existing orthographic transcripts using the same measure definitions. Its value lies in being a separately collected cohort, closer to the site-to-site transfer the clinical protocols demand.
The third experiment is the local-repair curve. For each external cohort we plot flag agreement against $k$ under three conditions: recalibration only, recognizer fine-tuning at equal labeling cost, and the full-cohort oracle. The decisive check is whether ten children suffice to close most of the frozen-to-oracle gap.
Finally, we test whether the label-free cohort signals rank the measures by their eventual calibration error across the external cohorts. A reliable ranking would give schools an advance warning before deployment. An unreliable one would argue for mandatory local transcription of a small sample at every new site. Both outcomes are informative.
References
- Can ASR generate valid measures of child reading fluency?
- Automated Estimation of Oral Reading Fluency During Summer Camp e-Book Reading with MyTurnToRead
- Estimation of children's reading ability by fusion of automatic pronunciation verification and fluency detection
- A Bayesian network classifier for word-level reading assessment
- Automatic assessment of children's reading level
- Automatic Assessment of Oral Reading Accuracy for Reading Diagnostics
- Prediction of Outcome in Acute Lower Gastrointestinal Bleeding Using Gradient Boosting
- Developing and validating COVID-19 adverse outcome risk prediction models from a bi-national European cohort of 5594 patients
- All models are local: time to replace external validation with recurrent local validation
- Radiomics in prostate cancer imaging for a personalized treatment approach - current aspects of methodology and a systematic review on validated studies
- Two methods for assessing oral reading prosody