Choosing Features by Between-Subject Evidence to Stop Identity Leakage in Small Cohorts

When a clinical dataset has 40 patients but 1,600 recordings, most machine learning pipelines treat it as 1,600 independent examples. Every feature-ranking statistic, saliency vote, or principal-component cut inherits that assumption. Features that are stable within a person but differ between people are exactly the ones that let a model recognize individuals, and the assumption makes them look highly predictive. Our hypothesis is that in small cohorts, this bias in feature selection, and not only the train/test split, is the main route by which identity leakage enters. We propose to score each feature or direction against the right noise level: the variation between subjects, not between recordings. We then keep only features whose task signal holds up at the level of whole people.

The opportunity matters because the field already knows the symptom and not the cure. Record-wise splitting produced 100% accuracy for autism classification from hand kinematics, which fell to 75% once whole children were held out (Cavallo et al. (2021)). A decade of age-estimation gains dissolved under subject-exclusive splits (Paplhám et al. (2024)). Subject-exclusive evaluation stops inflated numbers from being reported. It does nothing to stop the model from spending its limited capacity on fingerprints during training. A selection criterion that is cheap, closed-form, and correct about where the noise lives could recover real accuracy in exactly the regime where more data is unavailable.

Core Claim

Treat each feature $j$ as a nested measurement: a class effect, a subject-within-class effect (the "fingerprint"), and trial-to-trial noise. The ordinary trial-level test divides class variance by trial noise. It therefore rewards features with a large subject effect, because with only about 20 subjects per class, random subject differences leak into the class means and get counted as signal. The correct test for generalizing to new people divides by the subject-within-class mean square instead:

$$F^{\text{subj}}_j = \frac{MS_{\text{class},j}}{MS_{\text{subject(class)},j}},$$

where $MS_{\text{class},j}$ is the mean square of feature $j$ between classes and $MS_{\text{subject(class)},j}$ is its mean square between subjects within the same class. This is the textbook mixed-model F-test, and we do not claim the statistic is new.

The claim we will test is empirical and mechanistic. The standard selectors used in this literature systematically retain components with high intraclass correlation (the fraction of a feature's within-class variance that is due to who the subject is). These selectors are variance-ranked principal component analysis (PCA), trial-level correlation filters, and saliency voting from networks trained on trials. At a fixed number of retained dimensions, how well the retained features can identify subjects predicts the loss in held-out-subject accuracy. Swapping the denominator closes a meaningful part of the gap.

The same logic covers labels that change within a person, such as drowsy versus alert. There the correct denominator is the subject-by-class interaction, which is the repeated-measures analogue. To obtain directions rather than single features, we solve a shrunken generalized eigenproblem:

$$S_B w = \gamma \, (S_{S} + \epsilon I)\, w.$$

Here $S_B$ is the scatter of class means computed from subject means, $S_S$ is the scatter of subject means around their class mean (or the subject-by-class interaction scatter), $\gamma$ is the eigenvalue being maximized, and $\epsilon$ is a shrinkage constant chosen by nested cross-validation. We call the resulting procedure subject-level selection.

Research Program

The first strand is diagnosis. On each dataset we compute every feature's intraclass correlation and its trial-level and subject-level F statistics. We then ask which features each existing selector keeps. The selectors are top-variance PCA; the greedy eigencomponent search of Gula et al. (2023), which showed that variance is a poor proxy for task utility but ran its search on i.i.d. images; Pearson filtering; and the class-activation-map channel voting of Zhou et al. (2024), which aggregates saliency across subjects but scores from trial-level network confidence.

For each retained set we measure identity capacity. This is subject-ID decoding accuracy from held-in trials of training subjects, computed within class so that it is not just label decoding. We regress held-out-subject accuracy on identity capacity at fixed dimension. The file's own gap analysis poses exactly this question for PCA and leaves it open.

The second strand is the method: subject-level scoring and the eigenproblem above, applied to hand-crafted features, to EEG band-power and channel features, and to frozen deep embeddings. All selection statistics are computed strictly inside each training fold. Doing otherwise is a separate leakage channel that the file flags as unaddressed.

The third strand is a controlled synthetic benchmark. We plant $k$ class-signal dimensions and $f$ fingerprint dimensions with tunable intraclass correlation, vary the number of subjects from 10 to 250, and vary trials per subject. Ground truth lets us measure the rate at which each selector recovers the planted signal. It also lets us test whether the advantage of subject-level selection shrinks as subjects grow. That is the direction Ayotte et al. (2021) found for group leakage, where the gap only closed past 100–250 identities.

We will also examine the tension the file raises about repulsive diversity priors (Zou and Adams (2012)). Such priors spread components apart and gained most on small data, but they may spread components across individuals. We will measure the identity capacity of DPP-diversified components as a secondary analysis, not as part of the method.

Baselines

The naive reference point is no selection at all. We train the dataset's standard classifier on the full feature set under the same subject-exclusive folds. On MDDD this corresponds to the full-array InterpretableCNN, which Zhou et al. (2024) report at 78.26% leave-one-subject-out accuracy.

The strongest channel-selection competitor is interpretability-guided channel selection (ICS) from the same paper. It is a teacher network whose saliency votes pick the top 10 channels for a retrained student, and it is reported at 81.38% under leave-one-subject-out. The paper also reports that it beats Pearson and genetic-algorithm selection by about 2–2.5%. It is the strongest competitor because it already enforces subject-independent evaluation and already aggregates across subjects. We will reimplement it with channel voting done inside each fold, give it the same channel budgets, and use the same student architecture. Beating it means higher mean accuracy at matched $N$ and lower cross-subject standard deviation, with a paired test across held-out subjects.

On the depression EEG sets, the strongest reported pipeline is DiffMDD (Wang et al. (2024)). It reports 94.0% accuracy and 93.8% F1 on Mumtaz2016, and 80.0% accuracy and 79.2% F1 on Arizona2020, under 10-fold subject-independent cross-validation. We do not expect a linear selector to match a diffusion-augmented transformer outright. The fair comparison instead inserts subject-level selection in front of the DiffMDD classifier (or our reimplementation of it) and asks whether accuracy holds or improves while identity capacity falls. The folds, tuning budget, and augmentation are kept identical.

The greedy search of Gula et al. (2023) is run with nested held-out-subject accuracy as its criterion, and it receives the same compute budget as our eigen-solve plus its tuning.

The key ablation keeps everything in our method but swaps the denominator back to trial-level noise. Any gain that survives this swap is not due to our central idea.

Why not just…?

Why not just split by subject and regularize harder? Subject-exclusive splitting is required, and we use it everywhere. But Cavallo et al. (2021) show it only reveals the drop from 100% to 75%; it does not recover anything. Dropout and weight decay shrink all features uniformly and cannot tell a fingerprint from a biomarker. Our no-selection baseline with a tuned regularizer tests this directly.

Why not just z-score each subject's features? Per-subject normalization removes fingerprint offsets cheaply. When the label is constant within a subject, as in depression or autism, it also removes the between-subject class difference, which is where the signal lives. We expect it to fail on Mumtaz2016 and Arizona2020. When labels vary within a subject, as in drowsiness in MDDD, it is a real threat, and it additionally needs calibration data from each test subject. We will report it separately in both regimes. If it matches subject-level selection on MDDD, we will say our method adds value only in the constant-label regime.

Why not just train an encoder adversarially to be unable to predict the subject? With subject-level labels, the subject determines the class, so an unconditional subject adversary erases the class too. The class-conditional version, which predicts the subject only within a class, is the serious alternative. We include it with an equal tuning budget and will report its fold-to-fold variance, which we expect to be high at 27–64 subjects. If it wins, the conclusion is that learned invariance beats closed-form selection even at this scale, and we will state that.

Why not just run greedy selection with a held-out-subject criterion? This is the most direct threat, since it optimizes the right objective. With about 40 subjects, however, each greedy step picks the best of many candidates on a noisy validation estimate. That produces a winner's-curse bias toward components that happened to fit the validation subjects. The nested comparison, and the synthetic benchmark with known ground truth, will show whether this bias exists or whether greedy search simply wins.

Proposed Experiments

The datasets are Mumtaz2016 (64 subjects) and Arizona2020 (42 subjects) for the constant-label regime, and MDDD (27 subjects, leave-one-subject-out) for the within-subject-varying regime. We will also use the CMU keystroke benchmark (51 users). There we rerun the group-leakage setup of Ayotte et al. (2021) to test whether choosing timing features by impostor-level rather than repetition-level evidence recovers part of the 31–32% EER gap without adding impostors. MORPH with frozen pretrained face embeddings is a stretch case for dense within-subject labels (Paplhám et al. (2024)). If the kinematics data of Cavallo et al. can be obtained, it is the cleanest test. For keystrokes, no reported leakage-free EER value was found on Lacuna beyond the relative gaps.

The metrics are held-out-subject accuracy and F1 (or EER), cross-subject standard deviation, and within-class identity capacity of the retained features.

Three checks decide the question. First, across selectors and at matched dimension, identity capacity should be negatively related to held-out-subject accuracy, and top-variance principal components should show higher intraclass correlation than components chosen at the subject level. Second, subject-level selection should beat the trial-denominator ablation and ICS on MDDD at matched channel counts. Third, on the synthetic benchmark, its advantage should shrink as the number of subjects grows. If the first check fails, the mechanism is wrong and the method is at best a regularizer. If only the third fails, the story about why small cohorts are affected needs revision.