Order the Scan Only When It Shrinks the Interval: Conformal Biomarker Acquisition for Alzheimer's Progression

When a patient with mild memory complaints walks into a clinic, the doctor already has cheap information: age, cognitive test scores, the APOE ε4 genotype, and increasingly a plasma blood test. Expensive or invasive tests come later, if at all. These include structural MRI, amyloid PET, tau PET, and a lumbar puncture for cerebrospinal fluid (CSF). The useful question is not "how accurate is the best model with everything?" It is "for this patient, will ordering the next test make my forecast of their decline narrow enough to act on, and if I order it, can I trust the range I report?" This note proposes a method that answers both parts with a statistical guarantee. The method uses conformal prediction, a wrapper that turns any model's output into an interval guaranteed to contain the true outcome with a chosen probability (say 90%) under minimal assumptions.

The idea in two sentences: train a model that, from the cheap tests alone, forecasts how wide the calibrated interval for future decline would be under each possible set of additional tests, and order the cheapest set whose forecast width falls below a clinically actionable threshold. Because the decision depends only on information available before ordering, we can calibrate each decision branch separately on past patients who would have been sent down that branch, and that restores a coverage guarantee conditional on the decision that naive approaches lose.

Core Claim

Two lines of work meet here without touching. On the multimodal side, Flexible Multimodal Neuroimaging Fusion for Alzheimer's Disease Progression Prediction builds PerM-MoE, a mixture-of-experts model that gives each imaging modality its own routing network and substitutes learned placeholder vectors for missing scans. It predicts the 2-year change in Clinical Dementia Rating–Sum of Boxes (ΔCDR-SB) on ADNI and reports that tau PET is missing in over 87% of subjects. It outputs only point predictions. On the uncertainty side, Conformal Prediction with Missing Values shows that "impute, then conformalize" is valid on average but under-covers specific missingness patterns. It proposes calibrating within each pattern, or artificially masking richer calibration points down to the test pattern, but it treats missingness as something that happens to you, on low-dimensional tabular data, under a missing-completely-at-random assumption.

Our claim is that once missingness becomes a choice (the clinician decides which tests to order), the right object is a routing policy. Coverage should then hold conditional on the route taken, not merely conditional on the mask. These are different. Suppose the policy sends a patient to "add amyloid PET" because their cheap features look ambiguous. The calibration pool for the amyloid-PET mask then contains many easy patients the policy would never have routed there, so a mask-conditional interval can systematically under-cover exactly the hard patients who receive it. Calibrating per route fixes this, using past patients who have the route's modalities and whose cheap features would have triggered that route.

This also separates the proposal from Valid Selection among Conformal Sets. That paper selects the smallest set among several conformal models after seeing all of them at test time, and must pay for the selection bias with a stability-constrained randomized rule (MinSE) or an extra recalibration split. Here selection happens before the expensive data exist, so it depends only on the cheap features. Ordinary group-wise (Mondrian) calibration on the route label is then exactly valid, with no stability machinery. The cost is a new problem that paper does not face: predicting the width of an interval you cannot yet compute.

Research Program

Tiers and acquisition sets. We define tier-0 features available for everyone: demographics, MMSE, ADAS-Cog subscores, FAQ, and APOE ε4, roughly the ingredients that gave AUC 0.8111 without MRI in Sparse learning and stability selection for predicting MCI to AD conversion using baseline ADNI data. We then add plasma Aβ42/40 where available, T1 MRI, amyloid PET, and tau PET or CSF. An acquisition set $S$ is a superset of tier-0. Each set has a relative cost $c(S)$, and we sweep cost ratios rather than assert prices.

Predictor. The predictor is a PerM-MoE-style fusion model trained once on the training split with random modality dropout, so that it produces a lower and upper quantile of ΔCDR-SB for any mask. This is conformalized quantile regression (CQR), the interval method used in the missing-values paper.

Width forecaster. A separate small network $g(x_0, S)$ takes tier-0 features $x_0$ and a candidate set $S$. We train it on held-out training subjects who have $S$ observed: the target is the width of their conformalized interval after masking down to $S$.

Routing rule. The policy is

$$ r(x_0) = \arg\min_{S} \; c(S) \quad \text{subject to} \quad g(x_0, S) \le w^*, $$

where $w^*$ is a clinically actionable width, for example an interval that excludes "no decline". If no set meets $w^*$, the rule falls back to the cheapest set with the smallest forecast. Because $r$ depends only on $x_0$, the rule can be evaluated on every calibration subject.

Route-conditional calibration. For route $S$, the calibration pool is all calibration subjects $i$ with $r(x_{0,i}) = S$ whose observed modalities contain $S$. Each pool member is masked down to $S$, following the nested-masking trick of the missing-values paper. The conformal quantile is computed within that pool.

The threat is that in ADNI, PET and CSF were acquired preferentially for certain participants, so subjects with $S$ observed are not exchangeable with all subjects routed to $S$. If acquisition depends only on tier-0 features (missing at random given $x_0$), we can reweight calibration points by the inverse of their estimated probability of having $S$ observed given $x_0$. This gives approximate validity, with the approximation error governed by propensity-estimate quality. We will not claim guarantees under dependence on unobserved factors. Instead we stress-test it (see the experiments below).

Small route pools. Rare routes starve the calibration pool, the same failure that Uncertainty-Calibrated Prediction of Randomly-Timed Biomarker Trajectories with Conformal Bands reports for fine-grained Mondrian groups. We handle this by merging routes whose pool falls below a minimum size into the next-cheaper route with a larger pool. Merging never invalidates coverage because the merged label is still a function of $x_0$.

Second endpoint. A classification version uses future diagnosis (CN, MCI, AD) with prediction sets instead of intervals. Here "actionable" means a singleton set.

Baselines

The naive reference point is tier-0 only with global split conformal: everyone gets the cheap model and one global threshold, at zero acquisition cost. Any policy that spends money must produce narrower intervals than this at matched coverage, or it is pointless.

The opposite pole is acquire-everything: the full fusion model with all modalities, conformalized with the mask-conditional calibration of Conformal Prediction with Missing Values, restricted to subjects who have everything. This bounds achievable width from below at maximal cost.

The strongest existing competitor is PerM-MoE combined with CP-MDA-Nested, evaluated under the same routing policy but calibrated by mask rather than by route. It reuses the best published missing-modality fusion model and the best published missingness-aware conformal method, and differs from our proposal in exactly one ingredient. No reported interval width or coverage for ΔCDR-SB on ADNI was found on Lacuna for any method. The closest reported figures are point-accuracy ones. PerM-MoE improves RMSE over Flex-MoE by 8.6% with FLAIR alone and 4.3% with Aβ PET alone, and wins 11 of 15 modality combinations (Flexible Multimodal Neuroimaging Fusion for Alzheimer's Disease Progression Prediction). We will therefore first reproduce PerM-MoE's point accuracy on our split as a sanity check before comparing intervals.

A heuristic-acquisition baseline routes by the uncalibrated predictive spread of the tier-0 model, ordering more tests when the raw quantile gap is large, and then conformalizes by mask. This tests whether the learned width forecaster matters. A fixed clinical rule ("order amyloid PET for MCI with ambiguous plasma ratio") gives a policy a clinician might use without any model.

All methods share the same encoders, training split, calibration split, and hyperparameter search budget, and are run over at least 20 random calibration/test splits. We will count it as beating the strongest competitor if our method keeps route-conditional coverage within one percentage point of nominal on every route containing at least 30 test patients, where the mask-calibrated version misses by more on at least one route. We also require our method to match or reduce mean cost at equal fraction of actionable intervals. The key ablation swaps route-conditional for mask-conditional calibration and holds everything else fixed. A second ablation removes the propensity weights.

Why not just…?

Why not just calibrate per mask with CP-MDA-Nested and let the clinician read off widths? Mask-conditional validity is a real guarantee and is the closest prior method. It promises coverage averaged over everyone who has that mask, not over everyone a policy sends to that mask. Under any sensible policy those populations differ: the hard cases get more tests. This is a genuine threat, because if the tier-0 features carry little information about residual difficulty, the two populations nearly coincide and the gap vanishes. The ablation above measures the route-conditional coverage gap directly. If it is under a point everywhere, we will report that the simpler method suffices and the contribution reduces to the width forecaster and acquisition policy.

Why not just order tests where the model's own uncertainty is high? Uncalibrated uncertainty from trajectory models was shown to under- and over-cover unpredictably in Uncertainty-Calibrated Prediction of Randomly-Timed Biomarker Trajectories with Conformal Bands. More importantly, high uncertainty now does not mean a test will reduce it. Cognitive endpoints such as ADAS-Cog13 were no better than chance in the TADPOLE prospective evaluation (The Alzheimer's Disease Prediction Of Longitudinal Evolution (TADPOLE) Challenge: Results after 1 Year Follow-up), so some uncertainty is irreducible. The heuristic-acquisition baseline tests this: it should spend money on patients whose intervals do not shrink.

Why not just use an ensemble, as in TADPOLE where mean and median consensus won? Ensembling improves point forecasts but says nothing about which test to order or whether the reported range covers. We will use an ensemble as the base predictor if it helps, since the conformal layer is model-agnostic.

Why not a sparse linear model on tabular features? Biosignature-15 reached AUC 0.8587 and found that CSF added nothing significant (Sparse learning and stability selection for predicting MCI to AD conversion using baseline ADNI data). That is a serious possibility for our setting too: deep imaging features may not shrink intervals beyond tier-0. Our framework accepts a sparse model as the base predictor, and we will run it both ways. If the linear base model gives equal widths, the policy will correctly learn to order fewer scans, which is itself a useful clinical finding.

Proposed Experiments

The main experiment is a retrospective acquisition simulation on ADNI. We take subjects whose ΔCDR-SB is observed. For subjects with richer data we hide modalities and let each policy "order" them, so the realized interval under any route can be computed and checked against the truth.

We report three quantities as functions of $w^*$ and cost ratio:

  • coverage overall, per route, and per baseline diagnosis and APOE ε4 status;
  • mean interval width;
  • mean acquisition cost, and the fraction of patients receiving an actionable interval.

For diagnosis sets, we repeat the analysis on TADPOLE, training on D1 and testing on D4 prospective subjects (TADPOLE Challenge: Prediction of Longitudinal Evolution in Alzheimer's Disease). D4 is a genuine temporal shift, so we report whether coverage survives it rather than assume it does. TADPOLE's finding that CSF and DTI helped diagnosis gives a concrete prediction: the policy should route ambiguous MCI patients toward those modalities for the diagnosis endpoint far more often than for ΔCDR-SB.

Three checks decide whether the idea holds.

Route-versus-mask coverage gap. If mask calibration does not under-cover routed hard cases, the central claim fails.

Width-forecaster quality. We will report the rank correlation between forecast and realized width per route. If it is near zero, the policy is no better than the fixed clinical rule.

Selective-acquisition stress test. We calibrate on a full-modality subset chosen to mimic clinical ordering (preferentially impaired subjects) and test on a random subset, with and without propensity weights. This shows how much coverage the not-at-random acquisition in ADNI actually costs, which is the main assumption the method cannot prove away.