Medical imaging models often learn shortcuts: they predict disease from something that happens to travel with the label, such as patient sex, a hospital's scanner, or anatomy tied to demographics, instead of from the disease itself. Current fixes only work when someone has labeled the shortcut in advance. Many of the dangerous shortcuts, though, are unrecorded. This proposal starts from a fact about longitudinal data, meaning repeated scans of the same patient. Anything that stays fixed for a patient across visits, whether recorded or not, has no way to explain how that patient changes between visits. If a model is trained to explain within-patient change, stable confounders drop out. No confounder labels are needed.
The catch is that this cancellation only holds if the model combines the patient's fixed traits with the changing measurements by addition. A standard neural network mixes everything together, so a fixed trait can alter how a change is read, and subtracting one visit from another no longer removes it. Additive neural networks provide the missing structure. These are models that pass each input through its own small sub-network and add up the outputs. In such a model, subtracting visits cancels exactly the fixed-per-patient term. Additivity therefore stops being only an interpretability feature and becomes the assumption that makes label-free deconfounding work. We propose to build this model, test it where the shortcut is known, and use a sensitivity analysis to measure the confounders it cannot remove: those that change over time.
Core Claim
Suppose an outcome for patient $i$ at visit $t$ follows
$$y_{it} = u_i + \sum_{k} f_k(x_{itk}) + \varepsilon_{it}.$$
Here $u_i$ is an unobserved patient-constant term that absorbs sex, race, stable anatomy, and a fixed scanner or site. Each $x_{itk}$ is a regional imaging measurement (for example, a hippocampal volume or a regional PET uptake value). Each $f_k$ is a learned one-input shape function, and $\varepsilon_{it}$ is noise. Differencing two visits gives
$$y_{it} - y_{is} = \sum_k \big[f_k(x_{itk}) - f_k(x_{isk})\big] + (\varepsilon_{it} - \varepsilon_{is}),$$
which contains no $u_i$. The shape functions fitted from this equation therefore do not reflect any time-invariant confounder, measured or not. This is the logic of the econometric "fixed effects" design, carried into a nonlinear neural additive model. For binary labels, the same cancellation on the logit scale follows from the conditional likelihood used in fixed-effects logistic regression, where one conditions on each patient's number of positive visits.
The claim has two parts. First, a within-subject Neural Additive Model protects against time-invariant unobserved confounders, which existing imaging shortcut methods cannot do without labels. Second, the additive structure lets us state what it does not protect against: confounders that change over time, such as scanner upgrades or treatment devices that appear between visits. We bound that remaining risk explicitly instead of assuming it away.
Relation to Prior Work
Two lines of work motivate the gap. Benchmarking Dependence Measures to Prevent Shortcut Learning in Medical Imaging splits a network's internal features into task and confounder parts and removes their statistical dependence. On a CheXpert task where the training correlation between pleural effusion and sex is inverted at test time, this lifts AUROC from 45.8 to 87.7. Detecting shortcut learning for fair medical AI using shortcut testing uses gradient reversal to show that encoding an attribute is not the same as using it. Both methods require the confounder to be labeled during training, and both use single, static images. We give up the label requirement and pay for it with a different assumption: the confounder must be constant within a patient. Because the two assumptions differ, the two approaches complement each other rather than compete directly.
Hidden stratification causes clinically meaningful failures in machine learning for medical imaging supplies the failure our design cannot fix. A pneumothorax model that relies on the chest tube reaches 99% accuracy when the tube is present and 40% without it. A chest tube appears after diagnosis, so it varies over time, and differencing does not remove it. We use this as a negative control.
Additive architectures supply the method. Neural Additive Models introduced per-feature sub-networks. Don't PANIC combines such sub-networks with an image prototype network on ADNI (the Alzheimer's Disease Neuroimaging Initiative cohort). It uses baseline visits only, and its imaging branch has no protection against scanner shortcuts. SuperMAN and Neural Additive Vector Autoregression apply additive models to irregular time series. Both assume there are no hidden common causes, so their "faithful" attributions can faithfully report a confounded correlation. We keep the additive structure of these models, but we fit it to within-patient contrasts so that its attributions no longer carry fixed-per-patient bias.
Causal sensitivity analysis constrains how we report results. Sharp Bounds for Generalized Causal Sensitivity Analysis derives the tightest possible effect bounds as a function of a confounding-strength parameter $\Gamma$, including for time-varying exposures. We borrow that framing for the time-varying confounding that differencing leaves behind. We do not claim point identification. A perspective on individualized treatment effects estimation from time-series health data reviews latent-factor "deconfounders" whose factor assumptions cannot be tested. Our assumption, constancy within a patient, is narrower but can be checked directly on known confounders.
Research Program
The model has three parts, and they are added together. The dynamic part consists of shape functions $f_k$ over regional imaging measures and elapsed time. It is trained only on within-patient contrasts, or on the conditional likelihood for binary labels. The regional measures come from two sources. One is standard pipeline outputs, such as FreeSurfer volumes and regional PET uptake. The other is a small per-region image encoder that maps each region to one scalar, so the imaging branch stays additive. The PANIC prototype branch is not additive, and that is exactly where shortcuts can hide.
The static part is a separate head $g(s_i)$ over baseline information. It exists because a new patient's first scan still needs a prediction. It is trained after the dynamic part, on the residual. We report its share of each prediction as the "unprotected budget": the portion of the output that could come from time-invariant shortcuts.
The third part is the sensitivity layer. For each prediction of change, it reports the smallest strength $\Gamma$ of a time-varying hidden confounder that would reverse the prediction's clinical conclusion.
We will also test the additivity assumption itself, since the method depends on it. We fit a version that allows interactions between the static features and the dynamic features. The size of those interaction terms on held-out patients measures how badly the cancellation argument is violated in real data.
Baselines
The naive reference points come from carrying forward what is already observed. The first is the last observation: predict no change, or reuse the latest visit's label. CRONOS shows this "last context image" reference is hard to beat on slow-changing anatomy. It has an NRMSE of 8.38 on Lumiere, against 7.55 for CRONOS. The second reference is a baseline-only snapshot model. On ADNI, the strongest snapshot additive model is PANIC, which reports 60.7% balanced accuracy against 56.2% for the black-box DAFT model (Don't PANIC). We will retrain PANIC on our exact patient splits instead of quoting its number, because our task is longitudinal.
The strongest existing shortcut method is MINE-based feature disentanglement from Benchmarking Dependence Measures. It reports an inverted-CheXpert AUROC of 87.7 against 45.8 for an unprotected baseline. That figure is for a static detection task, so it does not transfer directly. We will rerun MINE on our longitudinal tasks with the same backbone features, data, and hyperparameter search budget. We will give it the true confounder label, which makes it an oracle our method does not get. Matching MINE-with-labels without labels counts as success. On the inverted test, "beating" it means our method stays within its confidence interval while using no confounder labels.
The strongest longitudinal baseline is the globally aligned enrichment approach from Learning Multi-Modal Biomarker Representations via Globally Aligned Longitudinal Enrichments. It reports an MMSE prediction RMSE of 0.2928, against 0.4188 for plain ridge regression, on 456 ADNI subjects. We will rerun it on our splits.
Two ablations isolate the key ingredient. The first is a black-box network trained on the same within-patient differences, which tests whether additivity matters or differencing alone does the work. The second is our additive model trained on pooled visits without differencing, which tests whether differencing matters or additivity alone does. The linear fair-representation method FR-CCA (Fair CCA for Fair Representation Learning), with a reported AUC of 0.773 and a demographic parity gap of 0.010 for sex on ADNI, serves as a labeled, linear reference point.
Why not just…?
Why not just train any network on difference images or change features? This is the most serious threat. If a black box trained on within-patient differences matches our model on the inverted-confounder tests, then additivity adds nothing beyond interpretability, and the core claim fails. Our reason to expect otherwise is structural. When a network is free to let a fixed trait change how a change is interpreted, differencing does not remove that trait's influence. When the confounder's effect is multiplicative in the input, the black-box-on-differences ablation should degrade under inversion while the additive model should not. The first ablation settles the question directly, and we will report it whichever way it comes out.
Why not just use a classical linear mixed-effects model? Random-intercept and fixed-effect linear models already cancel patient-constant terms, and they are the natural statistical baseline. What they cannot do is learn nonlinear shape functions, and the file shows that nonlinear trajectory shape carries prognostic signal. Association between albumin changes and prognosis in older patients with acute myocardial infarction reports an S-shaped relation between albumin change and survival. Linear debiasing can also lose accuracy: FR-CCA's AUC drops from 0.811 to 0.773 in Fair CCA. We run a linear fixed-effects model on the same regional features. If it ties with our model, the conclusion is that nonlinearity is not needed on ADNI, and we will say so.
Why not just label the confounders and use MINE or gradient reversal? When the confounder is known and labeled, that is a reasonable choice. We include it as the oracle baseline. Our method targets the case the file identifies as open: shortcuts nobody recorded. On held-out known confounders that we deliberately hide from our model, we measure how much of the oracle's robustness a label-free method recovers.
Proposed Experiments
The first experiment is a controlled injection on ADNI longitudinal MRI and PET. We add a synthetic intensity bias or watermark that is constant within each patient, correlate it with conversion from mild cognitive impairment to Alzheimer's disease in training, and invert the correlation at test time. We compare balanced accuracy and change-prediction RMSE against every baseline above. Because the confounder is constant within patients, the decisive expectation is that our model's accuracy barely moves under inversion, while pooled and snapshot models collapse, as the unprotected CheXpert baseline did at 45.8.
The second experiment uses a real, recorded confounder that we hide from the model. We use patient sex on ADNI, and, where CheXpert patients have repeated studies, the sex–effusion inversion from Müller et al. applied to predicting change in effusion status. We compare against MINE with access to the label.
The third experiment is the negative control. We inject a confounder that varies over time and appears only at later visits, mimicking the chest tube from Hidden stratification or a scanner upgrade. We expect differencing to fail here. The decisive check is whether the sensitivity layer assigns low $\Gamma$ robustness to the affected predictions and high robustness to unaffected ones.
The fourth experiment checks transfer to tumor imaging. On Lumiere glioma MRI, we test whether dynamic shape functions over regional tumor measures beat the last-observation NRMSE of 8.38 reported in CRONOS.
Across all experiments, we report the unprotected budget, the static head's share of each prediction, alongside accuracy. We also report the measured size of the static-by-dynamic interaction terms. If those interactions are large on real data, the additivity assumption fails in practice, and the claimed protection should be treated as approximate rather than exact.
References
- Benchmarking Dependence Measures to Prevent Shortcut Learning in Medical Imaging
- Detecting shortcut learning for fair medical AI using shortcut testing
- Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging
- Neural Additive Models: Interpretable Machine Learning with Neural Nets
- Don't PANIC: Prototypical Additive Neural Network for Interpretable Classification of Alzheimer's Disease
- SuperMAN: Interpretable and Expressive Networks over Temporally Sparse Heterogeneous Data
- Neural Additive Vector Autoregression Models for Causal Discovery in Time Series
- Sharp Bounds for Generalized Causal Sensitivity Analysis
- A Perspective on Individualized Treatment Effects Estimation from Time-series Health Data
- CRONOS: Continuous time reconstruction for 4D medical longitudinal series
- Learning Multi-Modal Biomarker Representations via Globally Aligned Longitudinal Enrichments
- Fair CCA for Fair Representation Learning: An ADNI Study
- Association between albumin changes and prognosis in older patients with acute myocardial infarction