The retina is wired from the same embryonic tissue as the brain and fed by small vessels that age the way brain vessels do. That is why a photograph of the back of the eye can flag silent strokes and predict dementia-related traits. Every eye-to-brain study so far, though, asks the brain a coarse question: did a stroke happen, or how large is a predefined region. This proposal asks a finer one. Given only a person's fundus photograph and retinal scan, how much of their gray matter map, measured point by point across the whole brain, can be predicted? And is that predictable part brain-specific, or just a readout of how fast the whole body is aging?
The design has three pieces. We use fast neural voxel-based morphometry to turn about six thousand UK Biobank brain scans into gray-matter density maps. We train a retina-to-map predictor with an objective that does not collapse onto the healthy average, because in a volunteer biobank the atrophy we care about is rare and spatially sparse. We then subtract what a non-neural aging clock can explain, built from automatically segmented vertebrae in the same participants' whole-body MRI. What remains is a map of where in the brain the retina carries information that the spine does not.
Core Claim
Retinal images predict a spatially specific, out-of-sample component of voxel-wise gray matter variation. This component survives adjustment for age, sex, blood pressure and a vertebral-marrow and body-composition aging score. It concentrates in regions known to be hit by small-vessel disease rather than spreading uniformly with global aging. It is recoverable only if training explicitly protects the rare, high-atrophy tail.
If the residual map is flat after the spine adjustment, that is also a clean and publishable answer. It would mean that current "retina sees the brain" results are largely generic systemic aging.
Why this is not already done
Eye-brain connections revealed by multimodal retinal and brain imaging genetics in the UK Biobank is the closest work, and it motivates the gap. It correlates 156 retinal traits with 458 brain MRI summaries in about 6,000 people and finds 625 associations, concentrated in the visual pathways. Adding retina to brain MRI raised fluid-intelligence prediction from $r=0.314$ to $r=0.344$. It works on parcel averages and handcrafted feature principal components. It never predicts voxel maps and never separates brain-specific from body-wide signal.
DeepRETStroke trains a large vision transformer on 895,640 fundus photos. It detects MRI-confirmed silent brain infarction at AUC 0.797, against 0.633 for clinical traits. The target is a binary label, not a tissue field.
Cerebral perfusion and gray matter volume alterations associated with cognitive impairment in metabolic syndrome with cerebral small vessel disease shows by voxel-based morphometry that small-vessel burden thins right middle frontal and left middle temporal gray matter. It has no retinal imaging and only 266 patients. We use its regions as an external anatomical prior, not as training data.
On the skeletal side, Pickhardt et al. show that automatically extracted trabecular bone and muscle density form a strong body-wide aging signal. Their model reaches a 10-year survival AUC of 0.880, against 0.779 for demographics. We borrow that idea, translated to MRI, as our systemic-aging control. Nobody has used vertebral phenotypes as a negative control for eye-brain specificity.
Research Program
Brain targets. Voxel-based morphometry warps each brain into a common coordinate space, labels each voxel as gray matter, white matter or fluid, and smooths the result. Every person then becomes a comparable 3D gray-matter density map. We run deepmriprep on all UK Biobank T1 scans with concurrent retinal imaging. That pipeline reports a median tissue Dice of 95.0, 4.6 s per scan against 173 s for CAT12, and statistical maps correlated $r>0.8$ with CAT12's.
On a random 300-scan subset we also run CAT12, to confirm that any retinal result is not an artifact of the neural pipeline's inherited biases. The target for each person is the deviation map: observed density minus a normative map predicted from age, sex and head size, fit on training folds only.
Systemic control. In participants who also have whole-body Dixon MRI, we apply MRSegmentator, an nnU-Net transferred from CT to MRI that segments 40 structures, to segment lumbar vertebral bodies. From the Dixon water and fat images we compute vertebral marrow fat fraction, alongside muscle and visceral fat volumes. Vertebral marrow is known to shift with metabolic and microvascular disease (MR-based proton density fat fraction (PDFF) of the vertebral bone marrow differentiates between patients with and without osteoporotic vertebral fractures MR-based proton density fat fraction (PDFF) of the vertebral bone marrow differentiates between patients with and without osteoporotic vertebral fractures; MR-Based Assessment of Bone Marrow Fat in Osteoporosis, Diabetes, and Obesity; MR-Based Assessment of Bone Marrow Fat in Osteoporosis, Diabetes, and Obesity). These measures, together with blood pressure, are combined into a body-aging score that contains no neural tissue. We report vertebral segmentation quality on a manually checked sample, because no vertebra-specific MRSegmentator Dice was found on Lacuna.
Retina-to-map predictor. The encoder is RETFound, a masked-autoencoder foundation model pretrained on 1.6 million fundus and OCT images. We fuse fundus and OCT tokens plus age and sex, following the metadata fusion in PRETI. The decoder outputs a gray-matter deviation map in a low-rank spatial basis: the top components of training-fold deviation maps, which keeps the output dimension tractable for about 6,000 subjects.
Tail-aware objective. The imbalance problem is concrete. Most participants are healthy volunteers, and most voxels in most maps are near zero. Plain mean-squared error therefore learns to predict the average map, the regression-to-the-mean failure seen when severe cases are scarce (Naimi et al.; Hong et al.). We adapt the inter-class mining idea of the Balance Loss, which in segmentation keeps the hardest background pixels in equal number to the foreground pixels. Here, for each subject, let $F$ be the set of voxels whose target deviation exceeds $\tau$ standard deviations, and let $H$ be the $|F|$ normal voxels with the largest current error. The loss is
$$L = \frac{1}{|F|}\sum_{v\in F}(\hat y_v-y_v)^2 + \frac{1}{|H|}\sum_{v\in H}(\hat y_v-y_v)^2 + \lambda \frac{1}{N}\sum_{v=1}^{N}(\hat y_v-y_v)^2,$$
where $y_v$ and $\hat y_v$ are the target and predicted deviation at voxel $v$, $N$ is the total voxel count, and $\lambda$ is a small weight on the full-map error. We also oversample subjects in the top decile of global deviation. Because hard-sample mining can amplify label noise (Balance Loss notes this risk at blurry boundaries), the threshold $\tau$ is applied to smoothed maps.
Vascular attribution. Kim et al. show that the retina still encodes age and sex after its vessels are digitally removed. We therefore retrain with vessels masked out, and separately with vessels only. This tests whether the brain signal is carried by the vasculature at all.
Baselines
Naive reference. The normative map itself, which predicts zero deviation for everyone. Every reported metric is computed on deviation maps, so this baseline scores zero explained variance by construction. Any positive held-out score means the retina adds something beyond age, sex and head size.
Strongest existing route. This follows the UK Biobank eye-brain study: predict regional volumes from retinal features, then paint the predicted parcel values back onto the atlas to produce a map. We run it with RETFound features rather than the original handcrafted features, with the same subjects, folds, covariates and hyperparameter budget. It is the strongest competitor because it is the published state of the art for eye-brain association and is far simpler.
The only reported figure on Lacuna is that study's fluid-intelligence gain, from $r=0.314$ with brain MRI alone to $r=0.344$ with retina added. No reported result was found on Lacuna for retina-to-voxel gray-matter prediction. Beating this baseline means significantly higher held-out voxel-wise explained variance, by paired bootstrap over subjects, inside small-vessel regions and in the high-atrophy tail.
Mass-univariate morphometry. The classical approach: regress each voxel's gray-matter density on a retinal age gap or on RETFound embedding components, then report a statistical map. This tests whether a learned decoder adds anything over standard statistics.
Ablations. Plain mean-squared error versus the tail-aware loss isolates the imbalance ingredient. Adjustment with and without the vertebral body-aging score isolates the specificity claim. The vessel-masked versus vessel-only encoders isolate the vascular claim. Every variant gets the same GPU budget and the same number of tuning trials.
Why not just…?
Why not just use regional volumes, as the UK Biobank study did? If retinal information about the brain is spread smoothly across large regions, the parcel baseline will match us. That would be a real result, and the baseline comparison above reports it directly. The reason to expect otherwise is that small-vessel atrophy in the metabolic small-vessel study was focal within gyri, and parcel averaging dilutes focal effects. Chen et al. also show that warped voxel fields keep individual-specific detail that averages discard.
Why not just run standard voxel-based morphometry with a retinal age gap as a covariate? This is the most serious threat. A single scalar may capture most of what retina predicts, and if so the deep decoder is unnecessary. We treat it as a full competitor rather than a strawman. If it matches the decoder, the proposal reduces to its other two contributions, the tail objective and the vertebral control, applied to that simpler pipeline. Both contributions transfer unchanged.
Why not just use clinical risk factors? Blood pressure, age and sex enter every model as covariates, so retinal information is measured beyond them by construction. DeepRETStroke already shows that retinal images beat clinical traits for silent infarction, at 0.797 against 0.633. The open question is not whether retina beats clinical scores. It is whether retina beats a body-wide imaging aging clock, which is why the vertebral score is the harder control.
Why not just train separate eye, brain and spine predictors? Separate predictors cannot answer the specificity question. The test is whether brain information in the retina exceeds what non-neural tissue explains, and that requires all three measured in the same people.
Proposed Experiments
Cohort. All UK Biobank participants with same-visit fundus, OCT and T1 imaging, roughly 6,000 per the eye-brain study. The vertebral analyses use the subset that also has whole-body Dixon MRI; we will report its exact size. Evaluation uses five subject-level folds with families kept together.
Primary metric. Held-out voxel-wise explained variance of the deviation map, averaged over gray matter. It is reported separately within the right middle frontal and left middle temporal regions from the small-vessel study, and within an occipital and visual-pathway mask, where the UK Biobank study found its strongest links.
Tail metric. Held-out explained variance restricted to subjects in the top decile of global deviation. This is where the tail-aware loss should help, and where plain mean-squared error should collapse toward zero.
Decisive checks.
- Specificity. After residualizing on the vertebral body-aging score, explained variance stays above zero, with confidence intervals excluding zero, and is enriched in small-vessel and visual regions relative to a size-matched random cortical mask.
- Tail objective. The tail-aware loss improves tail explained variance over mean-squared error without lowering whole-map explained variance by more than its bootstrap uncertainty.
- Vascular dependence. The vessel-masked encoder loses more signal in small-vessel regions than in visual regions, separating a vascular route from a retinal-ganglion-cell, visual-pathway route.
- Pipeline robustness. Results reproduced with CAT12 maps on the 300-scan subset correlate with the deepmriprep results.
Limitation. UK Biobank volunteers are healthier than clinical populations, a referral and selection effect described by Kujawa et al.. The file names no cohort with both retinal imaging and voxel-wise brain MRI in symptomatic patients. External clinical validation is therefore stated as future work, not claimed.
References
- Eye-brain connections revealed by multimodal retinal and brain imaging genetics in the UK Biobank
- A deep learning system for detecting silent brain infarction and predicting stroke risk
- Cerebral perfusion and gray matter volume alterations associated with cognitive impairment in metabolic syndrome with cerebral small vessel disease
- Biological age model using explainable automated CT-based cardiometabolic biomarkers for phenotypic prediction of longevity
- deepmriprep: voxel-based morphometry preprocessing via deep neural networks
- Segmenting Whole-Body MRI and CT for Multiorgan Anatomic Structure Delineation
- MR-based proton density fat fraction (PDFF) of the vertebral bone marrow differentiates between patients with and without osteoporotic vertebral fractures
- MR-based proton density fat fraction (PDFF) of the vertebral bone marrow differentiates between patients with and without osteoporotic vertebral fractures
- MR-Based Assessment of Bone Marrow Fat in Osteoporosis, Diabetes, and Obesity
- MR-Based Assessment of Bone Marrow Fat in Osteoporosis, Diabetes, and Obesity
- A foundation model for generalizable disease detection from retinal images
- PRETI: Patient-Aware Retinal Foundation Model via Metadata-Guided Representation Learning
- A Unified Loss for Handling Inter-Class and Intra-Class Imbalance in Medical Image Segmentation
- Effects of Hypertension, Diabetes, and Smoking on Age and Sex Prediction from Retinal Fundus Images
- Confirming the Diversity of the Brain after Normalization: An Approach Based on Identity Authentication