Removing Shared Information Before Fitting: Label-Free Orthogonalization for Multimodal Learning on ~100 Patients

Clinical prediction from speech, text, or images often has to work with about a hundred labeled patients. In that regime, adding a second informative feature set to a first one usually fails to help, and it often hurts. The second set mostly re-describes what the first already captured, and its leftover dimensions give the model new ways to overfit. On the ADReSS Alzheimer's speech benchmark, systems that searched over combinations of many feature sets reached 92.7% cross-validated accuracy but only 77.8% on the held-out test set (Tackling the ADReSS Challenge). We propose to remove the overlap between feature sets before any label is consulted. We fit a map from the dominant feature set to the secondary one on plentiful unlabeled recordings. We then give the small labeled learner only the part of the secondary features that this map cannot predict, compressed to a handful of dimensions.

The underlying observation is simple. Redundancy between two feature sets is a property of the inputs alone, so it can be estimated with thousands of samples even when labels number in the dozens. Current practice instead uses the scarce labels for two jobs at once: learning the task and deciding which features overlap. Recursive elimination, correlation-based selection, and grid search over subsets all work this way, and that selection step is where the optimism between cross-validation and test comes from. The same construction also gives a measurement. How much does the unpredictable remainder of B improve held-out loss once A is in the model? That quantity can be checked against realized test gains. It also speaks directly to an open dispute: do acoustic features carry anything beyond the pause information that can be written into a transcript as punctuation?

Core Claim

Let $A$ be the dominant feature set for a subject (for example, transcript features) and $B$ a secondary set (for example, acoustic descriptors). Let $\hat g$ be a regressor from $A$ to $B$, fit on an unlabeled pool much larger than the labeled cohort. The learner receives $A$ together with

$$\tilde B = P_k\big(B - \hat g(A)\big),$$

where $P_k$ projects the residual onto its top $k$ principal directions, again estimated on the unlabeled pool, with $k$ small (5–20). We will test three claims. First, at matched labeled sample sizes $N \in \{20, 50, 108\}$, a linear model on $[A, \tilde B]$ shrinks the gap between cross-validation and test and matches or beats the best label-selected fusion. Second, the cross-validated gain from adding $\tilde B$ predicts realized test gain across feature-set pairs better than the cross-validated gain from adding raw $B$. Third, the measured unique contribution of acoustics over pause-encoded transcripts is near zero with manual transcripts and grows when transcripts come from automatic speech recognition. That last prediction is falsifiable, and it settles the tension the file identifies between fusion papers and pause-encoding papers.

Relation to Prior Work

Three groups of papers motivate the gap by documenting diminishing returns from combining feature sets. Pate and Johnson show that syllable weight and dictionary stress carry essentially the same information for English word segmentation (Syllable weight encodes mostly the same information). Stenetorp et al. find approximate dictionary matching adds little once character n-grams are present (SimSem). Yousefpoori-Naeim et al. find that MPNet embeddings barely improve surface and cohesion features for USMLE item difficulty and do not help response time (Using Machine Learning to Predict Item Difficulty). All three diagnose redundancy after the fact, by ablation. None removes it in advance, and none uses unlabeled data to do so.

On ADReSS specifically, three systems motivate the gap from the small-sample side, and each handles redundancy with labels. The grid search over 16 feature sets (Tackling the ADReSS Challenge) and the RFECV compression of ComParE from 6,375 to 54 dimensions (Multiscale System) both select on the same labels they later evaluate with. INESC-ID concatenates x-vectors with linguistic embeddings into a linear SVM and finds acoustics nearly useless on their own, at 54.17% test accuracy (INESC-ID). Our method never uses labels to decide what is redundant.

Yuan et al. supply the strongest competitor and also partly constrain the proposal (Disfluencies and Fine-Tuning). They encode pauses as punctuation and fine-tune ERNIE, reaching 89.6% test accuracy. That result suggests acoustics may be almost fully redundant with pause timing. We take this seriously as a hypothesis rather than ignoring it, and we measure it.

Gradient-based redundancy control is the nearest methodological neighbor. RedReg suppresses gradient directions from a dominant modality that add little task gain (Adaptive Redundancy Regulation). RAML imposes sparsity on Gaussian unimodal features so a model stays robust when a modality is missing (Redundancy-Adaptive Multimodal Learning). Both act during labeled joint training, both were evaluated only on large benchmarks, and both depend on thresholds or penalties that are hard to tune without a reliable validation split. Our intervention happens before training and has one hyperparameter, $k$.

The geometric fusion head of MFM-Geom gains sample efficiency on 10% of PI-CAI by modeling cross-modal covariance (A Geometric Multimodal Foundation Model). It does not remove collinear blocks. We use it only as an optional secondary test bed.

Research Program

The first component is the orthogonalizer. The unlabeled pool for speech is built from three sources: the DementiaBank Pitt recordings with every ADReSS speaker removed (ADReSS derives from Pitt, so overlap must be checked carefully), the segment level of ADReSS training audio (4,076 normalized segments), and optionally CCC-style conversational audio. Labels are never read from this pool.

The roles are assigned as follows. $A$ is pause-encoded transcript features, either the ERNIE representation of Yuan et al. or cheaper character 4-grams and readability features, since 4-grams alone reached 86.4% in Tackling the ADReSS Challenge. $B$ is ComParE, x-vectors, or MFCC statistics. $\hat g$ will be ridge regression and, separately, a small multilayer perceptron, and we will report both. If a nonlinear $\hat g$ explains much more of $B$, that is itself evidence of redundancy that linear selection methods cannot see.

The second component is the measurement. For each ordered pair of feature sets $(A, B)$, we compute

$$\Delta(B \mid A) = \mathcal{L}(A) - \mathcal{L}(A, \tilde B),$$

the reduction in nested cross-validated log-loss when $\tilde B$ is added, with a bootstrap interval over subjects. The 16 feature sets of Tackling the ADReSS Challenge give 240 ordered pairs. We then ask how well $\Delta$ ranks the realized test-set gains, compared with the same statistic computed on raw $B$ and with the label-free share of $B$'s variance that $A$ explains.

The third component is a stress test of the punctuation-versus-acoustics question. We repeat the measurement with transcripts produced by automatic speech recognition. INESC-ID explicitly left this condition open.

A secondary replication uses the 667 USMLE items. There, $A$ is the surface and Coh-Metrix features, $B$ is MPNet, and $\hat g$ is fit at the sentence level across the item texts, with no difficulty labels. This tests whether the PCA-on-concatenation used by Yousefpoori-Naeim et al. left recoverable unique signal on the table.

Baselines

The naive reference is the ADReSS challenge linguistic baseline, at 0.75 test accuracy and 5.20 test RMSE for MMSE (ADReSS Challenge). It is joined by the single best modality alone, run through our own pipeline.

The strongest label-selected fusion baselines are the logistic regression found by grid search, reported at 77.8% test accuracy and 4.4388 MMSE RMSE (Tackling the ADReSS Challenge); the INESC-ID linear SVM fusion at 81.25% (INESC-ID); and phoneme-plus-audio late fusion at 79.17% (Multiscale System). We will rerun all of them inside the same nested cross-validation on the official 108/48 split. Each method gets the same budget of hyperparameter configurations, and every selection step, including RFECV, happens inside the inner folds.

The strongest existing method overall is ERNIE with 3-bin pause encoding and a 50-seed majority vote, reported at 89.6% ADReSS test accuracy (Disfluencies and Fine-Tuning). That paper reports no MMSE result, and the file has no reported result on Lacuna for the pause-encoded model under transcripts from automatic speech recognition. Beating it with manual transcripts would mean exceeding 89.6% with a linear head on $[A, \tilde B]$, where $A$ is the same ERNIE representation, at equal seed ensembling. The 48-subject test set moves in steps of about 2.1 points, so a single test-set win is not decisive. We will therefore also require a paired improvement in log-loss over repeated cross-validation whose bootstrap interval excludes zero. An honest null result here, meaning $\Delta \approx 0$, is a valid finding that supports Yuan et al.

Two ablations isolate the key ingredients. The first fits $\hat g$ and $P_k$ only on the labeled training subjects, which tests whether the unlabeled pool matters. The second replaces the residual with PCA of raw $B$ at the same $k$, which tests whether removing overlap matters beyond simply compressing. A random projection at the same $k$ serves as a floor. On USMLE, the baseline is the reported best test RMSE of 0.269 for difficulty and 23.271 seconds for response time (Using Machine Learning to Predict Item Difficulty).

Why not just…?

Why not just run RFECV, CFS, or Lasso inside nested cross-validation? These methods spend the scarce labels on deciding what is redundant, which is exactly where selection optimism enters. In Using Machine Learning to Predict Item Difficulty, feature rankings changed across algorithms that had identical RMSE, which suggests label-driven selection at this scale is unstable. Properly nested RFECV is still a real threat, so it is run as a matched baseline. The decisive comparison is the size of the cross-validation-to-test gap and the variability of selected features under resampling, both at $N = 20$ and $N = 50$, where we expect label-driven selection to degrade fastest.

Why not just fit ridge regression on the concatenation? Linear models were the best performers in Tackling the ADReSS Challenge, and ridge handles collinearity without discarding anything. This may well match our method at $N = 108$. The difference is that ridge shrinks every direction of $B$ equally, including the hundreds that only re-describe $A$ or encode speaker and channel. Our ablation of PCA versus residual at fixed $k$, swept over $N$, will show whether the advantage holds once labels are scarce. If ridge ties our method everywhere, we will report that the practical contribution reduces to the measurement $\Delta$.

Why not just drop acoustics and use pause-encoded transcripts? With manual transcripts this may be the right answer, and Yuan et al.'s 89.6% says so. The proposal treats that as the null hypothesis rather than arguing against it. What pause encoding cannot do is report what it discards, such as pitch and voice quality, or adapt when transcripts are noisy. The condition with transcripts from automatic speech recognition decides whether the residual acoustic signal matters in deployment.

Why not just use RedReg-style gradient gating? RedReg's monitor estimates task-gain growth from training dynamics. With 108 subjects and tiny batches those estimates are dominated by noise, and the file notes its thresholds are hard to tune without stable validation splits (Adaptive Redundancy Regulation). We will include RedReg on a small fused network as a secondary baseline at equal tuning budget. Our method avoids joint neural training altogether.

Decisive Checks

The proposal succeeds if three things happen. First, at $N \le 50$, the residual model's gap between cross-validation and test is reliably smaller than that of nested RFECV and ridge on concatenation, across 20 resampled subcohorts. Second, $\Delta(B \mid A)$ predicts realized test gains across the 240 feature-set pairs better than raw cross-validated gain, measured by Spearman correlation with a bootstrap interval. Third, the unique acoustic contribution over pause-encoded text rises measurably when transcripts come from automatic speech recognition.

The proposal fails informatively in two ways. If the ablation that fits only on labeled subjects performs as well as the full method, the unlabeled-pool claim is wrong. If the PCA ablation ties the residual model, removing overlap adds nothing beyond compression.