Speech as a Sequence of Residues: Lag-Structure Descriptors for Voice Biomarkers

Protein biologists predict what a protein does from its amino acid string alone, without ever solving its 3D shape. They do this by replacing each letter with a few measured properties (hydrophobicity, charge, volume) and then asking how those properties co-vary at fixed distances along the chain. The idea is that the order of residues quietly encodes the missing structure. This proposal applies the same move to clinical speech. A recording from a person being screened for Alzheimer's disease is converted into an ordered string of timed events: voiced stretches, pauses, fillers, and recognized words. Each event gets a handful of measured properties, such as duration, pitch, energy, and the frequency of the following word in everyday language. We then compute the same composition, transition, and lagged auto-covariance descriptors used for proteins and feed them to gradient-boosted trees. The bet is that cognitive decline shows up less in how often someone pauses than in where pauses fall relative to what comes next. A long pause before a rare word, repeated across a narrative, is a word-finding signature. Standard acoustic feature sets average this ordering away.

This matters because the field is stuck on two facts. Acoustic-only models lag transcript-based ones: on the ADReSS test set the best acoustic baseline reached 0.625 accuracy against 0.75 for linguistic features (Alzheimer’s Dementia Recognition Through Spontaneous Speech: The ADReSS Challenge). The strongest text models, in turn, depend on manual transcripts and on averaging dozens of unstable fine-tuning runs. Clinical cohorts are tiny, from 108 training subjects in ADReSS to 91 in SLaCAD. That is the regime where fixed, low-variance descriptors plus tree ensembles have historically beaten end-to-end neural models, in protein prediction as much as anywhere.

Core Claim

Order-aware, fixed-length descriptors of the speech event sequence carry diagnostic signal beyond marginal timing statistics. The descriptors we mean are lagged auto- and cross-covariances of event properties, borrowed from sequence-based protein interaction prediction. Gradient-boosted trees on these descriptors, computed from automatic speech recognition (ASR) and voice-activity output only, can close much of the gap between acoustic-only and manual-transcript models on small clinical cohorts.

The claim is falsifiable by one manipulation. We randomly permute event order within each recording. This preserves every marginal distribution (pause rate, mean voiced duration, word-frequency histogram) but destroys lag structure. If the permuted descriptors perform as well as the real ones, the idea is wrong.

The method comes from Multi-scale encoding of amino acid sequences for predicting protein interactions using gradient boosting decision tree. That paper maps sequences through seven physicochemical properties and extracts frequency, composition-transition-distribution, and auto-covariance descriptors, 347 dimensions per protein. With gradient-boosted trees it reports 95.28% accuracy on yeast interactions and beats random forests and support vector machines. We reuse this descriptor family and learner. The difference is that our "residues" are speech events and our "properties" are acoustic and lexical measurements. We add cross-property covariance, which the protein work does not emphasize but which is where our main clinical hypothesis lives.

The motivating evidence comes from speech. SLaCAD: A Spoken Language Corpus for Early Alzheimer's Disease Detection found that a single timing ratio, voiced duration relative to pauses, was among the strongest SHAP-ranked predictors of CSF tau/amyloid positivity. Disfluencies and Fine-Tuning Pre-Trained Language Models for Detection of Alzheimer's Disease showed that inserting binned pause tokens into transcripts lifts BERT/ERNIE to 89.6% on ADReSS, and that "uh" and "um" behave oppositely in patients. Both results say timing matters. Only the transformer result hints that timing in context matters, and it gets that context through a large model on manual text. We aim to capture the contextual part with explicit, inspectable lag statistics.

Research Program

Each recording is first turned into an event sequence. Voice-activity detection segments audio into voiced and silent stretches, and ASR with word timestamps labels voiced stretches as words or fillers. No manual transcript is used in the main condition. Each event $i$ receives a property vector $P(i)$ with a small fixed set of channels: log duration, mean pitch, pitch range, energy, event class (word, filler "uh", filler "um", pause), and, for words, log corpus frequency and a coarse part-of-speech group. Following the protein practice of combining qualitative groupings with quantitative indices, which Zhou et al. found beat either alone, categorical channels feed composition and transition counts, and continuous channels feed covariance terms.

The central descriptor is the lagged cross-covariance between property channels $j$ and $k$ at lag $d$:

$$ C_{jk}(d) = \frac{1}{n-d}\sum_{i=1}^{n-d}\bigl(P_j(i)-\bar P_j\bigr)\bigl(P_k(i+d)-\bar P_k\bigr), $$

where $n$ is the number of events in the recording, $P_j(i)$ is the standardized value of channel $j$ at event $i$, and $\bar P_j$ is its recording mean. Setting $j=k$ recovers the protein auto-covariance descriptor. The pre-registered clinical hypothesis is that $C_{\text{pause-dur},\,\text{word-rarity}}(1)$ is higher in impaired speakers, because they pause before hard-to-retrieve words. We restrict lags to $d \le 8$. As a secondary block, following innov'SAR, which applies a fast Fourier transform to per-residue property profiles, we compute low-order spectral power of the pause-duration series to capture rhythmic regularity. That work found the best property choices to be task-dependent. We treat this as a constraint and fix the channel set a priori rather than searching over it, since the cohorts are too small to survive descriptor search.

The resulting vector has a few hundred dimensions, similar to the protein case. SLaCAD reports that its tree models peaked at only eight selected features, and the research file flags greedy splitting degradation on large redundant sets. So every feature screen happens inside the cross-validation folds. The learners are LightGBM and random forests. Many lag descriptors are correlated, and the bagging variance floor $\rho\sigma^2$ discussed in the ensemble-tree literature argues for aggressive per-split feature subsampling. We tune that parameter explicitly.

Baselines

The naive reference point is a majority-class predictor, together with the published ADReSS challenge baselines: 0.625 test accuracy for acoustic features and 0.75 for manual-transcript linguistic features, with test RMSE of 6.14 and 5.20 for Mini-Mental State Examination (MMSE) regression (Alzheimer’s Dementia Recognition Through Spontaneous Speech: The ADReSS Challenge). Those baselines were run without tuning, so beating them alone proves little.

The fair acoustic competitor is therefore a tuned gradient-boosted tree on the openSMILE eGeMAPS and ComParE functional sets. These standard toolkits summarize frame-level acoustics with statistics like means and percentiles. The competitor gets the same folds, the same in-fold feature screening, and the same hyperparameter budget (an identical number of Bayesian-optimization trials) as our model. This is the most important comparison, because it is the strongest thing a practitioner would do without our idea.

The strongest marginal-timing competitor reproduces SLaCAD's recipe. It uses DiSVoice prosodic features, including the voiced-to-pause ratio, plus spaCy lexicosyntactic features from the same ASR output, with random forests, XGBoost, SMOTE, and top-feature selection. SLaCAD reports ROC-AUC of 0.72 for CSF tTau/Aβ42, 0.70 for Aβ42/Aβ40, and 0.71 for plasma pTau181 (SLaCAD). Those figures come from its own feature pipeline, and we rerun it under our folds.

The strongest existing method overall is Yuan et al.'s pause-token ERNIE ensemble at 89.6% ADReSS test accuracy on manual transcripts (Disfluencies and Fine-Tuning Pre-Trained Language Models for Detection of Alzheimer's Disease). We rerun it on our ASR transcripts with its forced-alignment pause bins. We report it both with its 50-run vote and with a compute-matched smaller vote, because ASR errors may change its standing.

The key ablation is the within-recording permutation described above. We recompute all descriptors after shuffling event order, keeping model and tuning fixed. A second ablation removes only the cross-property terms and keeps auto-covariance. This tests whether the pause-before-rare-word coupling, rather than generic temporal smoothness, carries the gain.

Beating a baseline means a paired improvement whose bootstrap confidence interval excludes zero over repeated cross-validation. On the 48-subject ADReSS test set, a single subject is about two accuracy points. We will not claim a win from a one- or two-subject difference, and we treat repeated cross-validation on the pooled 156 subjects as the primary evidence.

Why not just…?

Why not just tune a gradient-boosted tree on standard acoustic feature sets? This is the most serious threat. Hundreds of openSMILE functionals already include some rhythm and pause statistics, and part of the ADReSS acoustic shortfall may simply reflect untuned baselines. The research file contains no reported result for a tuned tree ensemble on eGeMAPS/ComParE for ADReSS. The matched baseline answers this directly, and we also report the concatenation of both sets. If eGeMAPS plus tuning matches our descriptors and the concatenation adds nothing, the contribution reduces to "tune your trees," and we will report that.

Why not just use a few interpretable timing summaries, as SLaCAD did? SLaCAD's strongest features were marginal ratios, and its models did best with eight inputs. It is plausible that all the signal lives in how much someone pauses, not where. The permutation ablation is built to settle exactly this, because it keeps every marginal summary intact. If permuted and unpermuted descriptors perform equally, the order-aware premise is refuted.

Why not just fine-tune a pretrained language model with pause tokens? Yuan et al. show it works well, at 89.6% on ADReSS. But the result uses manual transcripts, and the authors needed a majority vote over 50 fine-tuning runs to tame seed variance on 108 training subjects. We expect ASR errors to hurt a model that reads word identity more than descriptors built from timing and coarse word frequency, which tolerate misrecognitions. We do not assume this: the ASR-transcript rerun is the test. Our model may well stay below the manual-transcript transformer. The claim we care about is the transcript-free, compute-light regime.

Why not just pool embeddings from a pretrained speech encoder? Self-supervised audio embeddings capture local acoustics richly, but mean-pooling them discards ordering much as functionals do. No reported result was found on Lacuna for such embeddings on ADReSS or SLaCAD. We include pooled-embedding plus tree as an additional arm under the same budget, but we cannot cite a number.

Proposed Experiments

The primary experiment uses ADReSS: binary classification and MMSE regression, repeated stratified 10-fold cross-validation on all 156 subjects, and the official 48-subject test split reported once. The main metrics are accuracy, ROC-AUC, and RMSE. The decisive checks are two. First, our model must beat the tuned eGeMAPS/ComParE tree. Second, the permutation ablation must lose most of that margin. Both must hold for the core claim to stand, and we will also report per-descriptor SHAP values to check whether the pre-registered pause-to-word-rarity term is among the drivers.

The second experiment uses SLaCAD, where the targets are CSF and plasma biomarker positivity rather than diagnosis. With 63 and 77 labeled participants and very few impaired speakers, we use leave-one-out ROC-AUC with bootstrap intervals, compared against the rerun SLaCAD pipeline. This tests whether lag structure tracks biology in cognitively normal people, a harder and more useful question than separating diagnosed patients.

A third, smaller check trains on ADReSS and evaluates on SLaCAD's cognitive scores without retuning. This probes whether order-aware descriptors transfer across recording conditions (studio picture description versus Zoom interviews) better than frame-level functionals. We regard it as exploratory given the shift and the sample sizes.

The honest failure mode is that everything reduces to marginal pause rates. The design makes that outcome visible rather than hidden.