Label Error or Transitional Case? Auditing Severity Labels in Small Brain-Connectivity Cohorts

When a model grades disease severity in stages rather than giving a yes/no answer, a training label can disagree with the data for two very different reasons. Sometimes the label is wrong. Sometimes the patient really sits between two stages, for example someone labeled "healthy" who will be diagnosed with mild cognitive impairment at the next visit. Methods for cleaning noisy web-scale data cannot tell these cases apart. They flag whatever disagrees with its neighbors, so in a small clinical cohort they tend to delete exactly the rare in-between patients that fine-grained staging exists to capture. We propose an auditing procedure that separates the two cases. It watches how each subject's predicted severity moves as a deliberately simple model is given more capacity. Wrong labels are fit only once the model is allowed to become complex. Transitional subjects settle stably near a boundary between adjacent stages.

The setting is functional connectivity. Each subject's brain scan is summarized as a matrix of correlations between the activity time courses of brain regions, and the task is staging Alzheimer's progression, normal to mild cognitive impairment (MCI) to Alzheimer's disease (AD), from the Alzheimer's Disease Neuroimaging Initiative (ADNI) cohort. Three facts make this setting a natural test bed. Connectome benchmarks have so far stayed binary, and they report that severity prediction remains unsolved because of skewed classes (Data-Driven Network Neuroscience). Plain linear classifiers are as good as graph neural networks there. ADNI is longitudinal, which gives an outside check on which apparent "errors" were really early-stage patients.

Core Claim

In small connectivity cohorts with ordered stage labels, a regularized linear model traced along its regularization path can separate label errors from transitional subjects, and it does so better than neighborhood consensus or loss-based filtering. The regularization path is the sequence of solutions obtained as the penalty on weight size is relaxed from strong to weak. Correcting the errors while keeping the transitional subjects as soft, two-stage targets should improve held-out staging more than either discarding all disagreeing subjects or smoothing every label uniformly.

The mechanism comes from the "complexity barrier" of Symmetries of Functional Processes under Label Noise. That work shows clean patterns are fit early, at low functional complexity, while noisy labels require diverging complexity. It recovers clean concepts by averaging predictions over the training trajectory (its Concept Recovery Test) and across initializations (its Hypothesis Stability Test). The authors state that their complexity measure, neural persistence, does not scale to graph networks or transformers.

We change two things. First, we use a model whose complexity is just the size of its weight vector, so the barrier can be measured exactly and cheaply. Second, we add a third outcome their tests do not have. Their tests ask whether a label is clean or noisy. We ask whether it is clean, noisy, or genuinely between two adjacent stages. That third outcome only exists once the labels have an order.

Research Program

Each subject's connectivity matrix is computed with Pearson correlation and the Fisher z-transform over 90 regions of the AAL brain atlas, following BrainTF, and then vectorized. On this vector we fit a cumulative-link ordinal model. This model learns a single latent severity score $s_i(\lambda) = w(\lambda)^\top x_i$ for subject $i$ and two thresholds that cut the score into three stages. Here $x_i$ is the connectivity vector and $w(\lambda)$ is the weight vector at penalty strength $\lambda$. We trace the full path over $\lambda$, and we repeat it on $B$ bootstrap resamples that each exclude subject $i$.

For each subject this produces a path-integrated stage distribution,

$$\bar p_i(k) = \frac{1}{B}\sum_{b=1}^{B}\int p_b\big(y=k \mid x_i,\lambda\big)\, d\mu(\lambda),$$

where $k$ indexes stages, $p_b$ is the model fit on resample $b$, and $\mu$ weights the strongly regularized end of the path more heavily. We also record the fit point $\lambda_i^\ast$, the weakest regularization at which the subject's given label first becomes the model's top prediction.

The auditing rule reads three signatures from these quantities. A label error has $\bar p_i$ concentrated on a stage other than the given label, low variance across resamples, and a fit point reached only late in the path. A transitional subject has $\bar p_i$ split between the given stage and one adjacent stage, with its score stably near the threshold between them. A clean subject agrees with its label early in the path. Errors are relabeled to the stage $\bar p_i$ favors. Transitional subjects get a two-stage soft target equal to $\bar p_i$, in the spirit of the soft targets ALBEF's momentum teacher uses for partially matching web pairs (ALBEF). The final staging model is then refit on the audited labels.

As a secondary question, we will run the same audit on epoch-by-epoch trajectories of BrainNetCNN. This tests whether the complexity barrier also appears in a graph model on connectomes, which the label-noise paper leaves open.

The cited work plays three roles here. The benchmark paper's finding that linear models are competitive (Data-Driven Network Neuroscience) constrains the design: it is why the auditor is linear rather than a graph transformer. The complexity-barrier work supplies the method. LEMoN is the main foil. It scores a pair as mislabeled by combining direct embedding distance with nearest-neighbor agreement, and this works when the data is dense. Our claim is that it fails in the sparse regions where intermediate stages live. That failure is the open problem the field names as "elimination of atypical and rare concepts."

The multimodal text-anchored staging of CMCNet assumes curated, noise-free criteria templates. The geometric correlation head of MFM-Geom is evaluated only on binary prostate cancer. We deliberately do not import text alignment. ADNI provides no structured descriptor templates comparable to TI-RADS, and adding a text encoder would bring in capacity this sample size cannot support.

Baselines

The naive reference point is the same cumulative-link ordinal model, at a single cross-validated penalty, trained on the raw labels with no auditing. Because linear models are reported as competitive with graph networks on these cohorts (Data-Driven Network Neuroscience), this is a strong floor rather than a straw man. The benchmark reports accuracies between 60% and 70% across its ADNI and other binary tasks. We found no reported result on Lacuna for three-stage ordinal staging of ADNI from functional connectivity, so we will establish this number ourselves.

The strongest noise-handling competitor is LEMoN adapted to a single modality. It keeps the distance-plus-neighbor-consensus score but uses connectivity-vector neighbors and label agreement in place of image–text neighbors. It is the strongest competitor because it beat 12 baselines by more than 3% AUROC on classification and up to 6.3% on captioning (LEMoN). Its neighborhood size will be tuned with the same nested cross-validation budget as our path weighting $\mu$. We will also run a small-loss filter: discard subjects with high out-of-fold loss under the naive model.

The strongest staging model is BrainTF with its adapter (BrainTF), retrained with an ordinal head on the same folds. That paper reports gains over BrainNetCNN and BrainGNN on binary ADNI contrasts but gives no number we could find, so we will reproduce it ourselves.

Three ablations isolate the key ingredients. Uniform ordinal label smoothing gives every subject the same adjacent-stage softness and tests whether per-subject auditing matters. Bootstrap-only auditing at one $\lambda$ is the analog of the Hypothesis Stability Test alone. Path-only auditing without bootstrap is the analog of the Concept Recovery Test alone.

All methods share folds, preprocessing, atlas, and an equal number of tuning evaluations. To count as beating a baseline, the proposed method must improve held-out quadratic weighted kappa with a paired bootstrap confidence interval excluding zero at every training size we test. It must also match or exceed LEMoN's noise-detection AUROC while flagging substantially fewer true converters.

Why not just…?

Why not just fit regularized logistic regression and accept the noise? This is the baseline most likely to be "good enough," given the benchmark's finding that linear models match deep graph models. Our auditor uses the same model family, so any gain cannot come from capacity. If auditing does not beat the naive model, the conclusion is that label noise is not the bottleneck for connectome staging at these sample sizes. That outcome would be reported as a result in its own right.

Why not just use neighborhood consensus or small-loss filtering? LEMoN's own premise is dense embedding neighborhoods. In cohorts of fewer than 100 subjects (Bria et al.), a pre-converter MCI subject has few same-stage neighbors and will look like an error. The converter-retention experiment below measures this directly. If LEMoN retains converters as well as our method does, the three-way distinction is unnecessary and we will say so.

Why not just smooth every label toward its adjacent stages? This is the real threat. If inter-rater disagreement is spread roughly evenly across subjects, uniform smoothing captures most of the benefit for free. The ablation settles it. Per-subject auditing must beat uniform smoothing on kappa and on converter calibration, meaning that converters should receive higher adjacent-stage probability than stable subjects. Otherwise the per-subject machinery is not justified.

Why not just use a deeper connectome model like BrainTF? Its reported gains are on binary contrasts with no number available to us. The benchmark's plateau suggests capacity is not what limits performance here. We include it as a staging baseline and also run our auditor in front of it, which tests whether audited labels help a deep model as well.

Proposed Experiments

The primary data is ADNI resting-state fMRI, using the three-stage label at baseline. Training sets are subsampled to 60, 120, and 240 subjects to mirror small cohorts such as the 60-subject Mātai set, and test folds are held out by subject. Results are repeated with Schaefer parcellation, because the benchmark showed that the choice of atlas changes performance.

The first check uses synthetic noise. We flip 10–30% of training labels to an adjacent stage, the realistic noise pattern for ordered labels, and report detection AUROC for the injected flips against LEMoN, small-loss filtering, and the two single-component ablations.

The second check is the decisive one and uses real ADNI follow-up. We take subjects whose diagnosis changes at their next visit as ground-truth transitional cases. For each method we measure the fraction of these subjects flagged as label errors, where lower is better, and whether our method instead assigns them a two-stage soft target. If the trajectory-based audit cannot keep converters out of the error category better than neighborhood consensus, the core claim fails.

The third check is downstream staging after refitting on audited labels. We report quadratic weighted kappa, mean absolute stage error, and balanced accuracy.

A confound check correlates the error flags with head-motion estimates from preprocessing. Motion produces spurious correlations across the whole matrix, so it could masquerade as label noise; flags that track motion would indicate the auditor is detecting artifacts rather than mislabeling.

Finally, the BrainNetCNN trajectory experiment reports whether noisy subjects are fit later in training than clean ones. This gives a direct, cheaply measured test of the complexity barrier on graph models, which the original authors could not run.