Anchored Fundus-to-OCT Synthesis for Home Monitoring Between Eye Injections

People with diabetic macular edema (fluid leaking into the central retina) receive repeated eye injections. Whether the next injection is needed depends mostly on optical coherence tomography (OCT), a clinic scanner that produces depth-resolved cross-sections of the retina. A surface color photograph of the back of the eye, called a fundus photo, largely cannot show that fluid. Phone-based fundus cameras could move monitoring into homes and community clinics, but they record only surface images, and those images are blurred, jittery, and unevenly lit. Generative models can already turn clean clinic fundus photos into plausible OCT, but this is a badly underdetermined guess: a single 2D photo does not contain the depth structure. We propose to make the guess much less underdetermined by conditioning the generator on two inputs, the patient's own last real clinic OCT and today's noisy phone photo. The generator then only has to infer what changed since the last scan.

The core idea fits in two sentences. Treat the last clinic OCT as an anatomical anchor and the cheap phone photo as evidence of change since then, and train a model to synthesize today's OCT from both. Then use disagreement among several sampled synthetic scans as the rule for when the patient must come in for a real scan. The aim is a monitoring loop whose missed-fluid rate can be measured and controlled. Without it, phone screening cannot scale safely.

Core Claim

Anchoring cross-modal synthesis on a prior same-patient scan should turn fundus-to-OCT translation from an ill-posed reconstruction into a better-posed change-estimation problem. We expect three consequences. First, detection of new or worsening fluid on synthetic OCT should improve substantially over unanchored synthesis. Second, it should improve over carrying the last scan forward. Third, it should stay robust when the fundus input is degraded to handheld quality. A second claim is that spread across diffusion samples (a diffusion model is a generator that learns to produce images by gradually removing noise from random input) gives a referral signal. That signal should keep false negatives at a target rate while sending fewer patients to the clinic than fixed-interval visits. We do not claim that synthetic OCT contains information absent from its inputs. The value has to come from the anchor and from the paired-data prior, and the experiments are designed to show whether it does.

Relation to Prior Work

The closest synthesis work is the APTOS-2024 challenge report. It benchmarked fundus-to-3D-OCT generation on 1,247 paired exams. The best team used a latent diffusion model conditioned on retinal foundation-model features and reached a Fréchet Video Distance of 624.59. All inputs were clinic bench-top photos, there was no prior scan, and the organizers note that outputs are sparse representative B-scans rather than dense volumes. We reuse its architecture family as our starting point. We differ in adding a longitudinal anchor, targeting handheld inputs, and judging outputs by a clinical decision rather than distributional realism.

Two longitudinal generators supply method ideas but stay within high-end imaging. SADM is a diffusion model with a temporal transformer that generates future scans within one modality and handles irregular visit gaps; it reports SSIM 0.851 on cardiac MRI. Longitudinal Causal Image Synthesis links tabular clinical variables to future brain MRIs through a causal graph. Both need a full scanner acquisition at every observed timepoint. We borrow SADM's handling of irregular intervals, and our new observation comes from a different, cheaper modality.

On the consumer-hardware side, RVD supplies 635 handheld smartphone fundus videos and shows the size of the domain gap. A vessel model trained on clean bench-top data falls to 28.31% mIoU on handheld video. This motivates our design and constrains it, because a synthesizer trained only on clean photos should be expected to break. SpeCamX shows that software alone can pull clinically useful signal from an unmodified phone camera (R > 0.90 for bilirubin). However, it outputs a scalar from surface tissue, not subsurface structure.

The longitudinal ophthalmic datasets supply the data. OLIVES links fundus photos, OCT slices, and 16 biomarkers across 96 eyes followed for about 66 weeks through anti-VEGF trials. OCT4DME provides pre- and post-injection OCT from 2,000 patients with fluid labels.

Research Program

We would build a conditional latent diffusion model whose inputs are three things: the prior OCT volume at visit $t_0$, a fundus image at visit $t_1$, and the interval $t_1 - t_0$. Its output is the OCT B-scans at $t_1$.

Training proceeds in three stages. In the first stage, we pretrain unanchored fundus-to-OCT synthesis on APTOS pairs. In the second stage, we pretrain an anchor-only change prior, prior OCT plus interval to next OCT, on OCT4DME's pre- and post-injection sequences. In the third stage, we fine-tune the full anchored model on consecutive-visit pairs from OLIVES, where both fundus and OCT exist at each visit.

Robustness to handheld capture comes from a degradation model. We fit motion blur, jitter, and illumination drift to the statistics of RVD frames, and apply it to clean training fundus inputs. Because the model only needs to explain change, we also add a consistency term: when the fundus evidence is uninformative, the output should stay close to the anchor.

At deployment, the model draws $K$ samples. A fixed OCT fluid classifier scores each sample for intraretinal and subretinal fluid. Across those scores, the mean is the predicted fluid status and the spread is the uncertainty. A patient is referred when either the predicted probability of new fluid or the spread exceeds thresholds. We set those thresholds by split conformal calibration (a distribution-free way to pick cutoffs that bound the miss rate on held-out data) at a target sensitivity.

The fixed fluid classifier is trained on real OCT4DME scans and never sees synthetic data. That paper reports SRF/IRF detection AUC above 93% on real OCT. Using a fixed classifier means synthetic scans are judged by whether a real-OCT reader would reach the same decision. This follows the warning in the abdominal synthetic CT comparison that PSNR and SSIM do not track clinical consequences.

The main data limitation is that no public dataset pairs phone fundus with same-day OCT. Bench-top results with simulated degradation are therefore a proxy. The decisive test needs a small prospective collection: phone fundus captured at routine OCT visits, with a target of roughly a hundred eyes over several visits, at a site already doing injections.

Baselines

The naive reference is carry-forward. It predicts that today's fluid status equals the status on the last real OCT, adjusted by an interval-dependent recurrence rate fitted on training data. This is what a clinic implicitly assumes between visits. Any phone-based method that cannot beat it adds cost without value.

The strongest existing synthesis method is the APTOS-winning approach: a latent diffusion model on foundation-model fundus features, without an anchor. We will retrain it on the same APTOS-plus-OLIVES data with the same degradation augmentation and the same GPU budget and hyperparameter search budget as our model. Its reported challenge result is FVD 624.59 (APTOS-2024). That number was measured on clean inputs with a different evaluation protocol, so it is context, not a target. Beating it means higher sensitivity for new fluid at matched referral rate on the same held-out OLIVES eyes.

The strongest non-generative competitor is a direct discriminative model. It receives exactly the same inputs as our model: prior OCT features, today's fundus, and the interval. It predicts new or worsening fluid directly, uses a deep ensemble for uncertainty, and gets the same conformal calibration. It is trained with matched compute and tuning.

As a reference for what real OCT achieves on the related decision, OCT4DME reports AUC 80.06% for predicting continued treatment need. That figure is an upper bound with a full scan, not a competitor. For the value of structured intermediate representations, OLIVES reports 82.33% balanced accuracy with biomarker intermediates versus 70.15% from OCT alone. That evidence uses real biomarker labels, so it supports our approach but does not prove that synthetic intermediates help.

Three ablations isolate the ingredients. Removing the anchor reduces the model to the APTOS-style baseline. Removing the fundus gives a SADM-style anchor-only extrapolation, which tests whether the phone photo contributes anything. Removing degradation augmentation tests how much handheld robustness comes from the noise model.

Why not just…?

Why not just train a classifier to predict fluid change directly from the phone photo and prior scan? This is the real threat. Information-theoretically, a direct model given the same inputs can match anything computed from a synthetic scan. If the matched direct ensemble equals our model on sensitivity at fixed referral rate, we will say so plainly. The contribution then shrinks to two things: synthesis as a pretraining objective that exploits paired fundus-OCT volumes without fluid labels, and an output a clinician can inspect. We will test that pretraining role separately by fine-tuning the direct model from the synthesizer's encoder. The case for synthesis rests on label efficiency, because paired volumes outnumber fluid-labeled change events. The experiment that settles it is a learning curve in the number of labeled change events.

Why not just carry the last OCT forward, or extrapolate it over time with a longitudinal model like SADM? If fluid recurrence were predictable from time since injection, phone imaging would be pointless. OCT4DME reports suboptimal treatment response in between 31% and 65% of patients, which suggests trajectories are heterogeneous. Still, the anchor-only ablation measures this directly. If adding the fundus photo does not improve on it, the phone component has no value. That would be a useful negative result for home monitoring programs.

Why not just fine-tune on clean photos and rely on users capturing good images? RVD shows that models trained on clean bench-top images collapse on handheld captures. The no-augmentation ablation and the prospective phone set measure how much this matters for synthesis specifically.

Why not use physics-based reconstruction instead of a generator? SpeCamX's Wiener-estimation approach works for spectral absorption at the tissue surface. It has no route to depth structure behind the retinal surface, so we do not consider it a competitor for this target.

Proposed Experiments

The primary experiment uses patient-level cross-validation on OLIVES consecutive-visit pairs, with fundus inputs degraded by the RVD-fitted model. The primary metric is sensitivity for new or worsening fluid at a referral rate fixed to match carry-forward's clinic load. Secondary metrics are the miss rate at the conformal target, fluid-classifier agreement between synthetic and real OCT, and per-B-scan realism. Realism is reported only as a sanity check, given known metric discordance. We also check robustness by sweeping degradation severity.

The decisive checks are three. First, the anchored model must beat both carry-forward and the unanchored APTOS-style model on the primary metric. If it does not, the anchoring idea fails. Second, its margin over the matched direct ensemble decides whether synthesis is the mechanism or merely a pretraining aid. Third, the conformal miss-rate bound must hold on the prospective real-phone set, not only on simulated degradation. If it holds only in simulation, the RVD-fitted noise model is the gap to fix before any home deployment.

A hallucination audit accompanies all three. Retina specialists, blinded to source, grade sampled synthetic scans for invented or erased fluid. We report how often high sample spread coincides with those errors, which tests whether the referral signal catches the failures that matter.