Reluctant Residual Fusion for Speech-Based Dementia and Voice-Disorder Screening

Speech is one of the cheapest ways to screen for neurological disease: a one-minute picture description or a sustained "aaah" can be recorded on a phone, with no needle, scanner or specialist in the room. Most models built on such recordings, however, look excellent on the data their authors tuned them on and much worse on anything new. On the ADReSS Alzheimer's speech benchmark, a pipeline chosen by grid search over 16 feature sets reached 92.7% cross-validated accuracy and then 77.8% on the blind test set (Tackling the ADReSS Challenge). An x-vector acoustic model fell from 72.73% to 54.17% (INESC-ID). The more flexible components a pipeline contains, and the more combinations its builders try, the more it absorbs quirks of the development set. Those components are also the parts a clinician cannot audit.

We propose a fusion model that defaults to a small, transparent score built from clinically meaningful speech measurements: pause rate, lexical diversity, readability, disfluency counts, and for voice, standard perturbation and harmonicity measures. A deep audio or text network is added only as a correction to that score's errors. It stays in the model only if it passes a permutation test that is run under the same search procedure used to build the model. The bet is that one design delivers two things. Each prediction can be read as "clinical score plus a bounded, separately reported correction." And the gap between development and test performance shrinks, because the admission test charges the deep branch for the search effort spent on it.

Core Claim

Two ideas exist separately in the file. The reluctant-modeling principle of RAITR fits a sparse linear model first, lets nonlinear spline terms explain only what remains, and found that ordinary K-fold cross-validation over-admits those nonlinear terms. That work is limited to tabular covariates. The stop-gradient residual decomposition of Beyond Additive Fusion (MRO) makes higher-order fusion branches model only what lower-order ones leave unexplained, but it was tested only on sentiment and emotion benchmarks and has no protection against overfitting to the development set.

Our claim is that these ideas become useful for clinical speech only when a third element is added: an admission rule that is robust to meta-overfitting. Meta-overfitting is the leakage that comes from repeatedly choosing features, architectures and hyperparameters against the same small development set. Concretely, the predicted log-odds for subject $i$ is

$$\hat{s}_i = g_\theta(z_i) + a \cdot h_\phi\big(u_i,\ \mathrm{sg}[g_\theta(z_i)]\big),$$

where $z_i$ are the interpretable speech measurements, $g_\theta$ is a sparse logistic model, $u_i$ is the raw audio or transcript fed to a deep network $h_\phi$, $\mathrm{sg}$ is a stop-gradient so the deep branch cannot reshape the transparent score, and $a \in \{0,1\}$ is the admission decision.

The branch is admitted ($a=1$) only if its out-of-fold reduction in log-loss exceeds the 95th percentile of a null distribution. To build the null, we shuffle $u$ across subjects and rerun the entire branch-construction procedure, including every architecture and hyperparameter search we performed. A deep branch tuned through many trials therefore has to beat a null that was tuned through the same number of trials.

Research Program

We first build the transparent layer for two settings.

For Alzheimer's detection on ADReSS and ADReSSo, $z$ contains readability indices and number of unique words (as in Tackling ADReSS), pause and duration statistics, and disfluency counts (following Rohanian et al.). For ADReSSo, these are computed from automatic transcripts, since that challenge provides no manual ones (ADReSSo).

For the cross-cohort voice benchmark assembled by FairPDA, which classifies healthy controls, Parkinson's and ALS from phone and headset phonations across four cohorts, $z$ contains classical voice-quality measures.

The deep branches are a speech-encoder or ResNet audio model and a transformer text model. Each is trained on the frozen residual of $g_\theta$. Cross-fitting is grouped by subject throughout. On the voice benchmark, it is also grouped by recording cohort, so that the admission test asks whether the correction transfers across rooms and devices. That is exactly the failure FairPDA describes as learning "the acoustics of the room."

Three analyses accompany the model. First, we measure how much of the development-to-test drop each pipeline shows, and whether the admission decision predicts that drop. Second, we probe the admitted residual branch for cohort and device identity. If a linear probe recovers recording source from the residual's hidden layer much better than from $z$, the correction is carrying confounds. Third, we report per-subject audit statistics: the fraction of decisions that the residual flips relative to the transparent score, and whether those flips are concentrated in the ambiguous middle band of $g_\theta$.

Interpretability of the transparent layer will be checked the way Haddada et al. did, by computing a Kendall tau between the model's feature ranking and a ranking elicited from clinicians. Their tree models reached 1.0 on tabular cognitive features. Our features are speech measures, so expert rankings must be collected fresh, and we treat this as a secondary measure.

Baselines

The naive reference points are the official ADReSS challenge baselines: 0.625 test accuracy for acoustic features and 0.75 for linguistic features (Luz et al.). The ADReSSo late-fusion baseline reports 78.87% test accuracy (ADReSSo). We also include the transparent model $g_\theta$ alone, which is the "do nothing beyond the interpretable score" option and the one we most need to beat.

The strongest published deep fusion competitor on ADReSS is the gated highway fusion of Rohanian et al., which reports 0.792 test accuracy and 4.54 MMSE RMSE. INESC-ID reports 81.25% test accuracy with fused x-vectors and BERT features. Tackling ADReSS reports 86.4% for character 4-grams alone, but the file does not make clear whether that figure is cross-validated or on the test set, so we treat 81.25% as the firm test-set bar.

On the voice benchmark, the strongest method is FairPDA itself, combining MixStyle, partial adversarial domain alignment and gender debiasing. No reported numerical result was found on Lacuna for FairPDA's external MCC or balanced accuracy, so we will rerun it ourselves.

Every competitor is re-implemented on identical subject-grouped folds, with the same audio encoder where applicable and an equal hyperparameter-trial budget. A frozen protocol is registered before the blind test set is touched.

The key ablations isolate each ingredient. The first is the same residual model with $a$ chosen by plain cross-validation instead of the permutation-calibrated test, which isolates the admission rule. The second is joint training without the stop-gradient, which isolates the residual constraint. The third is the MRO-style decomposition with no admission rule at all.

Beating the field means two things. Test accuracy and AUC must match or exceed the best competitor's within confidence intervals. The development-to-test gap must also be smaller, with non-overlapping bootstrap intervals across repeated resplits. We will report intervals throughout because 48 test subjects make one subject worth about two points.

Why not just…?

Why not just use the transparent speech-feature model alone? This is the real threat. If the admission test rejects every deep branch, the honest result is that the transparent model is the right screening tool, and our method has correctly said so rather than shipping an overfit correction. The experiment settles this directly. If $g_\theta$ alone matches the 81.25% INESC-ID test accuracy, the deep branches add nothing at this data scale. That outcome is a finding, not a failure, and it fits the pattern in Haddada et al., where simple models on clinical features beat imaging. The evidence for a residual that earns admission comes from Rohanian's gain from fusion, 0.72 to 0.792, though that comparison was not controlled for search effort.

Why not just fix the evaluation protocol with nested cross-validation and touch the test set once? Good hygiene reduces optimistic reporting but does not stop a flexible model from fitting confounds that are shared across folds, such as one cohort's microphone. Nested cross-validation also does not make the model auditable. We include a nested-cross-validation "pick the best pipeline" baseline under the same trial budget. If it closes the generalization gap as well as our admission rule does, the gain is purely interpretability, and we will say so.

Why not just train a strong joint fusion model and explain it with SHAP? Post-hoc attributions are unreliable when features are collinear, and speech measures are heavily collinear. Applied to a joint model, they also do not constrain what the model learns. The no-stop-gradient ablation, explained with SHAP, gives the comparison. We will report whether its attributions are stable across resplits and whether it shows a larger development-to-test gap.

Why not just use domain adaptation as FairPDA does? Adversarial alignment can remove channel information, but it leaves a black box and does not address search-induced overfitting. We will also test FairPDA's alignment applied inside our residual branch. If that combination is what wins on held-out cohorts, the two approaches are complementary rather than competing.

Decisive Checks

The proposal succeeds if three conditions hold. First, on ADReSS and ADReSSo, the admitted model's development-to-test drop is smaller than that of grid-searched and gated-fusion pipelines while its test accuracy is not lower. Second, on leave-one-cohort-out voice classification, the admission rule rejects residual branches whose hidden representations encode recording cohort, and admits only corrections that improve held-out-cohort MCC. Third, admission decisions agree with test-set benefit more often than plain cross-validation decisions do.

The proposal fails if permutation-calibrated admission and plain cross-validation choose the same branches with the same test outcomes. In that case, the reluctant structure is ornamental.