Remote speech recordings are becoming one of the fastest ways to measure decline in diseases such as amyotrophic lateral sclerosis (ALS). Pause and timing measures taken from a read passage registered a clinically meaningful change in 8.5 weeks for bulbar-onset ALS. The standard clinical speech rating needed an estimated 242 weeks to register the same change (Multimodal Digital Biomarkers for Longitudinal Tracking of Speech Impairment Severity in ALS). These pause measures, however, come from an automatic detector that must decide where speech stops and starts. Detectors that classify sounds correctly are often much worse at saying when those sounds begin and end, and the gap grows under noise. On car-cabin surveillance sounds, timing-aware accuracy fell about 2 points below plain accuracy in clean audio and nearly 12 points below it at 0 dB signal-to-noise ratio (An acoustic event detection framework and evaluation metric for surveillance in cars). Over a year of home recordings, patients change phones, rooms and microphone distance, and caregivers begin to help. If the detector's boundary errors change along with those conditions, part of the measured "decline" is the measuring tool drifting, not the patient.
This note proposes to measure each session's boundary error directly and carry it into the longitudinal model. The measurement works by planting pauses and repetitions of known length into that session's own recording, using its own background noise, and checking how far the detector misses them. The longitudinal model then treats each session's timing measure as an observation whose bias and noise depend on that measured error. The deliverable is a progression slope, and a time-to-meaningful-change estimate, that is corrected for drift in the detector's accuracy. It also tells us when such a correction actually matters.
Core Claim
The claim is that in multi-month remote monitoring, boundary-localization error is not random noise that averages out. It is a session-level quantity that can be estimated without manual labels and that is correlated with calendar time within a patient. Ignoring it biases progression slopes and makes time-to-meaningful-change estimates look either too optimistic or too pessimistic. Correcting for it with a per-session "timing reliability" score should give slopes that match human-annotated slopes more closely than the uncorrected estimate does, or than a model that simply adjusts for signal-to-noise ratio.
The link to the longitudinal-tracking literature is direct. Clinical work repeatedly finds that short observation windows hide what happens later. Examples are complications missed after hospital discharge (Feng et al.), effects that disappear past a 12-week trial (Su et al.), and relapse cycles longer than a year (Xu et al.). Speech biomarker studies are responding by lengthening follow-up, from the roughly three-week test–retest window of Voiceome to the 12-month ALS cohorts. Longer windows also give recording conditions more time to drift. The timing-accuracy gap is therefore not a fixed nuisance. It is a confounder whose size grows with the very follow-up length that the clinical literature calls for.
Research Program
The program has three parts.
The first part is a per-session timing-reliability estimate, built by probe injection. For each recording session we take the patient's own audio and locate low-energy background stretches. We then synthesize short sets of probe events and insert them at known positions. The probes are silent pauses of controlled duration and sound or word repetitions made by splicing the patient's own syllables, following the forced-alignment repetition simulation of Frame-Level Stutter Detection. We run the production detector and score its boundaries with the time-dependent accuracy cost of Transfeld et al. Under that cost, boundary offsets below a small tolerance $\Delta t_1$ count as correct, offsets between $\Delta t_1$ and $\Delta t_2$ get partial credit, and offsets above $\Delta t_2$ count as errors. The output for session $j$ of patient $i$ is a reliability score $q_{ij}$ and a signed mean boundary offset $b_{ij}$, which can show, for example, that the detector systematically ends pauses late in this room.
We deliberately do not probe prolongations or silent blocks. Frame-Level Stutter Detection found that synthetic prolongations transfer poorly. SEP-28k notes that blocks carry little radiated acoustic energy and need articulatory features. Synthetic probes of those event types would measure the simulator rather than the detector. This restriction suits the ALS use case, because the most responsive biomarker there is pause and timing based.
The second part is a measurement-error-aware growth model. Let $y_{ij}$ be the observed timing biomarker, for example total pause time per passage, for patient $i$ at time $t_{ij}$. We fit
$$ y_{ij} = (\beta_0 + u_{0i}) + (\beta_1 + u_{1i})\,t_{ij} + \gamma\, b_{ij} + \varepsilon_{ij}, \qquad \varepsilon_{ij} \sim \mathcal{N}\big(0,\ \sigma^2 + \tau^2 (1-q_{ij})\big). $$
Here $u_{0i}$ and $u_{1i}$ are patient-specific random intercept and slope, and $\beta_1$ is the population progression slope. The parameter $\gamma$ maps the measured boundary bias into biomarker units; for pause duration it should be close to the number of pauses per passage, which gives a built-in sanity check. The parameter $\tau^2$ is extra noise that grows as reliability drops. The model reduces to the growth curves used in the ALS digital-biomarker study and the mixed-effects models of Listener effort measures clinically meaningful change of dysarthria in amyotrophic lateral sclerosis when $\gamma = \tau = 0$, so every comparison with them is nested. From the fitted slope and its uncertainty we recompute the time-to-meaningful-change, using the anchor-based minimal clinically important difference procedure of the 2024 ALS study.
The third part is a semi-synthetic longitudinal benchmark with known ground truth. Real cohorts lack millisecond annotation of every session. We therefore build trajectories with known true slopes by taking dysfluency clips from UCLASS, FluencyBank and SEP-28k, together with the hand-annotated aphasia and dyslexia set used by Towards Hierarchical Spoken Language Disfluency Modeling. We stretch pause durations according to a prescribed per-"patient" decline and mix the audio with background noise whose level and character drift over simulated months. That drift can be independent of time, correlated with time, or correlated with severity, the last mimicking sicker patients recording in worse conditions. The benchmark lets us check slope bias exactly. The real-cohort analyses then test whether the correction behaves the same way where the truth is only partly known.
Baselines
The naive reference point is the uncorrected growth model on the same timing biomarker, which is what the 2024 ALS study fits. The number to beat on its own terms is its report that reading-passage timing detects meaningful change in 8.5 weeks for bulbar-onset ALS, against 242 weeks for the clinical speech subscore. Our proposal does not aim to shorten 8.5 weeks. It aims to say whether that 8.5 weeks survives once detector drift is removed. A corrected estimate that is longer but agrees better with human-annotated trajectories would count as a success. On the semi-synthetic benchmark, beating this baseline means a lower absolute error in recovered slope, plus confidence intervals with close to nominal coverage, under the time-correlated and severity-correlated drift conditions.
The strongest existing longitudinal method is the Listener Effort prediction model. It is a Lasso regression on speaking rate, articulation rate and Whisper confidence that reaches $R^2 = 0.83 \pm 0.07$ against speech-language pathologist ratings and yields an ALS slope of 0.77 points per month, with detectable progression in 92% of patients. We run it on the same sessions with the same mixed-effects specification and the same tuning budget. We report whether adding $q_{ij}$ and $b_{ij}$ to its features changes its slopes. The concern is that Whisper confidence itself depends on recording conditions, so this model may carry the same drift.
The strongest detector baseline is Stutter-Solver. It treats dysfluency detection as one-dimensional object detection with start and end timestamps, and reports a time-based F1 of 0.893 on UCLASS against 0.700 for the previous best. A second detector is the hierarchical aligner of Lian and Anumanchipalli, which reports word-level F1 of 73.1% against 64.0% for Whisper-X on real disordered speech. Each detector is run with and without the correction, so we can see whether a better detector makes the correction unnecessary.
There are two key ablations. The first replaces the probe-based $q_{ij}, b_{ij}$ with session signal-to-noise ratio and device metadata as covariates. The second uses probe-based reliability only in the variance term ($\gamma = 0$), which separates bias correction from down-weighting of unreliable sessions. No reported result was found on Lacuna for any of these baselines on a drift-controlled longitudinal benchmark, because that benchmark does not yet exist.
Why not just…?
Why not just add signal-to-noise ratio as a covariate in the mixed model? This is the most serious competitor, and the program is built so that it can win. Noise level is only one cause of boundary error. Reverberation, microphone frequency response, codec artifacts and voice changes caused by the disease also shift where a detector places onsets. Transfeld et al. show that timing loss depends on noise, but their setup varies noise alone, so it does not show that noise level captures everything. The signal-to-noise ablation on both the semi-synthetic and real data settles the question. If it recovers slopes as well as probe-based reliability does, we will recommend the cheaper covariate, and the remaining contribution is the benchmark and the evidence that drift matters.
Why not just use a more accurate detector such as Stutter-Solver and skip the correction? A better detector shrinks the error, but drift in the error is what biases slopes, and a detector trained on podcasts and synthetic VCTK variants may drift in its own way on home recordings. The disordered-speech results already show a clear drop from synthetic to clinical audio. We test this directly. If Stutter-Solver's per-session reliability shows no correlation with time within patients, the correction is unnecessary for that detector, and we will report that outcome.
Why not just collect more sessions so that errors average out? Averaging removes random error but not systematic error that trends with time. The precondition check described below measures whether reliability trends within patients. If it does not, more sessions are indeed the right answer and the variance-only ablation should match the full model.
Why not just track Whisper word-deletion rates? Whisper is trained to output clean text and drops repetitions and hesitations. Word error rate also does not score boundary placement at all, as On designing and evaluating speech event detectors and Improved performance evaluation of speech event detectors argue. Whisper-based features remain a baseline here rather than being dismissed.
Proposed Experiments and Decisive Checks
The first check is whether probe injection is valid at all. On the hand-annotated disordered-speech set and a manually re-timed subset of SEP-28k clips, we test whether probe-estimated reliability predicts the detector's true boundary error on the natural events in the same clip. This is evaluated by rank correlation across clips, under the noise conditions taken from car-cabin and household recordings. Frame accuracy is reported only alongside event-level timing metrics, because Condron et al. show that frame accuracy can look near-perfect while event boundaries fail. If probe reliability does not track natural-event error, the program stops here.
The second check is the precondition. In whichever ALS cohort we can access through collaboration, either the 3,350-session cohort or the Everything ALS Speech Study, we fit a mixed model of $q_{ij}$ against time within patient. We also test whether that within-patient trend differs between bulbar-onset and non-bulbar-onset groups. A null trend means the bias story is wrong for these cohorts, and only the variance term can matter.
The third experiment is slope recovery on the semi-synthetic benchmark across the three drift regimes. The metrics are slope bias, interval coverage and time to detect a prescribed change, compared across the naive model, the signal-to-noise covariate model, the variance-only model and the full model, with three detectors.
The fourth experiment is real-cohort agreement. Speech-language pathologists annotate pause boundaries on a stratified sample of sessions, chosen to over-represent sessions with low reliability. We ask which model's patient-level slopes best match slopes computed from these human-timed pauses. We also report how the 8.5-week detection time and the bulbar versus non-bulbar slope gap (1.66 versus 0.42 points per month for listener effort) change after correction. The result that would change practice is a corrected time-to-meaningful-change that differs meaningfully from the uncorrected one while agreeing better with human timing. A result showing no difference would also be useful, because it would show that current remote timing biomarkers are robust to detector drift.
References
- Multimodal Digital Biomarkers for Longitudinal Tracking of Speech Impairment Severity in ALS: An Investigation of Clinically Important Differences
- An acoustic event detection framework and evaluation metric for surveillance in cars
- Osteoporosis exacerbates perioperative complications in total knee arthroplasty: a 10-Year nationwide analysis
- Comparison of add-on medications for persistent storage symptoms after α-blocker treatment in BPH patients – a network meta-analysis
- B cell and anti-PLA2R antibody-guided rituximab therapy in idiopathic membranous nephropathy: a prospective multi-center cohort study in the East Coastal Region of China
- A Longitudinal Normative Dataset and Protocol for Speech and Language Biomarker Research
- Frame-Level Stutter Detection
- SEP-28k: A Dataset for Stuttering Event Detection from Podcasts with People Who Stutter
- Listener effort measures clinically meaningful change of dysarthria in amyotrophic lateral sclerosis
- Towards Hierarchical Spoken Language Disfluency Modeling
- Stutter-Solver: End-to-end Multi-lingual Dysfluency Detection
- On designing and evaluating speech event detectors
- Improved performance evaluation of speech event detectors
- Non-Verbal Vocalisation and Laughter Detection Using Sequence-to-Sequence Models and Multi-Label Training