Deep models can read stroke and heart risk from photographs of the back of the eye. This works because the retina's small blood vessels are a developmental and anatomical extension of the brain's. The same patients, however, are usually on drugs that act directly on blood vessels: antihypertensives, statins, and diabetes drugs. A drug can widen retinal arterioles or stabilize leaky capillaries without changing the brain to the same degree. When that happens, a treated patient's retina looks healthier than their brain is, and a retinal risk model quietly under-reads them. No current retinal model checks for this.
We propose using the brain itself as the reference. In UK Biobank, several thousand people have both retinal images and brain MRI. The idea is to train the retinal representation so that its "brain-proxy" part carries no information about treatment beyond what the measured brain state already explains. The model that tries to detect treatment is therefore given the brain MRI and the pre-treatment covariates. This is the key design choice. The obvious fix, stripping all treatment information out of the image, also strips out disease, because doctors prescribe treatment according to disease. The two-sentence version: drugs can make the retina look better than the brain it is supposed to stand in for, and paired MRI lets us learn which parts of retinal appearance are drug effects and which are brain signal. We build a retinal risk model that uses only the brain-tracking part, and we measure, drug class by drug class, how much each treatment distorts the proxy.
Core Claim
Current retinal models predict cardiovascular and cerebral outcomes directly from images. Examples include RETFound, a masked-autoencoder foundation model pretrained on 1.6M retinal images (Zhou et al.); PRETI, which adds age and sex tokens and pairs of left and right eyes to pretraining (Lee et al.); and DeepRETStroke, which distills an MRI-confirmed silent-infarct detector into a retinal model (DeepRETStroke). These models treat retinal appearance as a fixed readout of vascular state. Our first claim is that the readout is treatment-dependent.
We call the retina proxy-faithful for treatment $T$ if a retinal feature $f(R)$ depends on treatment only through the shared microvascular state $V$ that also drives the brain. In symbols, $f(R) \perp T \mid V, C$, where $C$ is the set of pre-treatment covariates (age, sex, blood pressure, glycemic history). We measure $V$ through brain MRI markers $B$, so the quantity we estimate is the proxy gap:
$$\Delta_T = \mathbb{E}[f(R) \mid T{=}1, B, C] - \mathbb{E}[f(R) \mid T{=}0, B, C].$$
A nonzero proxy gap means that, at equal brain state, treated retinas look systematically different from untreated ones.
Our second claim is methodological. A retinal representation trained under a conditional independence penalty stays predictive of brain state and outcomes while becoming stable across treatment groups. An unconditional penalty, such as the gradient-reversal de-confounding of Adversarial De-confounding in ITE Estimation, cannot achieve both.
This differs from the closest connecting work in the file in three ways. The GLP-1 receptor agonist study (GLP-1 RA and DR progression) compares observational cohorts on retinal endpoints only, with no brain reference and no learned representation. Mendelian randomization analyses (Lu et al.; Eye-brain imaging genetics) use germline variants as instruments, which capture lifelong liability and are blind by construction to drugs started in adulthood. Causality matters in medical imaging supplies the causal picture we adopt: disease causes image features, and treatment adds its own path from treatment to image. That paper stops at causal graphs; we turn them into an estimator and a training objective on retina–brain data.
Research Program
The backbone is a pretrained RETFound encoder applied to fundus photographs, with optical coherence tomography (OCT) as a second arm. Its embedding is split into a brain-proxy block $z_B$ and a residual block $z_O$.
The brain-proxy block is trained to predict brain MRI markers $B$ in the UK Biobank paired subset of 6,454 individuals reported in Eye-brain imaging genetics. The markers are white matter diffusion and integrity metrics, thalamic volume and other regional volumes that study linked to retinal thickness, and white matter lesion burden where available. The MRI features come from the pretrained brain networks in UKBiobank_deep_pretrain together with standard image-derived phenotypes.
A treatment discriminator receives $z_B$ together with $B$ and $C$. It tries to predict each medication class (antihypertensive subclasses, statins, glucose-lowering drugs including GLP-1 receptor agonists, and antiplatelets). The encoder is trained through gradient reversal to defeat it. Because the discriminator already sees brain state and covariates, it can only succeed by exploiting treatment information in excess of what brain state explains. The encoder is therefore pushed to remove drug artifacts, not disease.
We keep the orthogonality penalty between blocks from Chauhan et al.. We also replace a single adversarial weight with a conditional kernel independence penalty on residuals. This addresses the instability in the balancing weight that the same paper reports. At deployment, the outcome model uses only $z_B$ and needs neither MRI nor medication records.
The same machinery gives the proxy-gap estimate $\Delta_T$ for each drug class. We fit $f(R)$ as a frozen RETFound feature or as interpretable vessel caliber from the vessel segmenters in SGL-Retinal-Vessel-Segmentation. We then adjust for $B$ and $C$ with doubly-robust estimation. Finally, a sensitivity analysis asks how much measurement error in $B$ as a stand-in for $V$ would be needed to explain the gap away. That error is the main threat, and we treat it explicitly rather than assume it away.
Baselines
The naive reference is RETFound fine-tuned directly on outcomes with no treatment handling. The published ceiling for this family in UK Biobank is a 3-year incident heart failure AUROC of 0.794 from fundus and 0.799 from OCT (Zhou et al.). For a composite of stroke, myocardial infarction, and chronic kidney disease, Lu et al. report a 10-year AUC of 0.737 from fundus plus age and sex, and 0.767 with clinical factors added. That 0.767 model, re-implemented on our split, is the strongest existing predictive competitor. It is the most accurate retinal composite predictor in the file and already uses clinical data.
The second baseline is the medication-as-covariate model: the same encoder with medication indicators and $C$ concatenated into the outcome head. The third is unconditional gradient-reversal de-confounding in the style of Chauhan et al., where the discriminator sees only $z_B$. This third baseline is the ablation that isolates our key ingredient, conditioning the discriminator on $B$ and $C$. The fourth is a clinical-only risk model; Lu et al. report PCE at AUC 0.726 and QRISK3 at 0.729.
All models share the same UK Biobank participants, splits, backbone, and number of hyperparameter search trials. We report a GPU-hour budget for each. Our model beats the baselines if it matches the 0.767-style model in overall AUROC within its confidence interval while reducing the treated-versus-untreated calibration gap. We also require it to degrade less than the covariate model when medication records are withheld or a drug class is unseen in training.
Why not just…?
Why not just add medication flags as inputs? This is the real threat for prediction, and it may match us when records are complete and a drug's retinal effect is roughly additive. The file's argument against it, the TLearner PEHE of 2.5233 versus 0.4008 in Chauhan et al., comes from a synthetic effect-estimation benchmark and does not settle retinal risk prediction. We therefore test the covariate approach directly. Our claimed advantages are narrower. First, we produce a proxy-gap estimate the covariate model cannot give. Second, we stay robust when records are missing or a new drug appears. The held-out-drug-class experiment decides the second point, and if the covariate model survives it we will say so.
Why not just remove all treatment information from the embedding? Treatment is prescribed because of disease, so any feature that predicts disease also predicts treatment. An unconditional adversary must erase disease signal to win. We expect this baseline to lose outcome AUROC, and the ablation measures exactly this.
Why not just restrict to untreated patients? This violates the requirement that both groups be represented: it discards most high-risk patients and yields an atypical cohort of undiagnosed people, as argued via Causality matters in medical imaging and the treated-heavy AlzEye population. We still report a treatment-naïve model as a sanity reference.
Why not just image the brain? MRI is the anchor during training precisely because it is too costly for screening. The retinal model needs no MRI at deployment. Clinical-only baselines are included, and they already trail retinal models: an AUROC of 0.633 versus 0.797 for silent infarction in DeepRETStroke.
Proposed Experiments
The first experiment is semi-synthetic and has known ground truth. Using vessel segmentation, we apply controlled arteriolar dilation to UK Biobank fundus images to simulate a drug. The simulated treatment is assigned with probability rising in the true MRI marker, which reproduces confounding by indication. We then check whether each method recovers the injected proxy gap and keeps brain-marker prediction intact. Conditional de-confounding should remove the injected effect while unconditional de-confounding also removes true signal. If it does not, the idea fails.
The second experiment uses real UK Biobank data. We compare outcome AUROC and calibration slope separately in treated and untreated strata, against the baselines above. We also run leave-one-drug-class-out training, and a condition in which medication fields are withheld at test time.
The third experiment reports $\Delta_T$ per drug class with sensitivity bounds for measurement error in the MRI anchor. As a targeted check of the unresolved tension in Kim et al., we ask how much of the drop in retinal-age $R^2$ from 0.92 to about 0.74 in hypertension and diabetes disappears when retinal age is computed from $z_B$ within drug-stratified groups. External transfer to AlzEye is limited to calibration by treatment-indicating diagnoses, because that dataset has no MRI.
For MRI-marker prediction from retinal images specifically, no reported result was found on Lacuna, so that baseline number will be established here.
References
- A foundation model for generalizable disease detection from retinal images
- PRETI: Patient-Aware Retinal Foundation Model via Metadata-Guided Representation Learning
- A deep learning system for detecting silent brain infarction and predicting stroke risk
- Adversarial De-confounding in Individualised Treatment Effects Estimation
- The Effect of Glucagon-like-Peptide-1 Receptor Agonists on Diabetic Retinopathy Progression, Central Subfield Thickness, and Response to Intravitreal Injections
- Phenotypic screening and genetic insights for predicting major vascular-related diseases using retinal imaging
- Eye-brain connections revealed by multimodal retinal and brain imaging genetics
- Causality matters in medical imaging
- AlzEye: longitudinal record-level linkage of ophthalmic imaging and hospital admissions of 353 157 patients in London, UK
- Effects of Hypertension, Diabetes, and Smoking on Age and Sex Prediction from Retinal Fundus Images