Where to Predict in Retinal OCT? Anatomy-Guided Target Selection for I-JEPA

Yunchu Feng, Wai Tak Lau, Kaveri A. ThakoorarXiv 2026 · arXiv

TL;DR

Researchers found that guiding the "masking" process in self-supervised learning with retinal anatomy significantly improves a model's ability to detect glaucoma from OCT scans. Surprisingly, a simple method using image intensity to locate retinal tissue performed nearly as well as complex, segmentation-based approaches, suggesting that basic anatomical awareness is a powerful tool for medical AI.

What Problem Does This Paper Address?

In self-supervised learning (SSL), models learn by solving "pretext" tasks on unlabeled data. A popular architecture, I-JEPA (Image-based Joint-Embedding Predictive Architecture), learns by masking parts of an image and trying to predict their latent representations from the surrounding context.

Standard I-JEPA places these masks (targets) randomly and uniformly across the image. However, in Retinal Optical Coherence Tomography (OCT), the actual retinal tissue occupies only a small, horizontal band. Uniform masking often ends up asking the model to predict "empty" space (the vitreous or background), which isn't very useful for learning medical features. This paper asks: Can we improve I-JEPA by forcing it to predict regions that actually contain retinal anatomy?

Methods

The authors investigated five different "masking families" to guide where the model should focus its predictions:

  1. RANDOM (Baseline): Standard uniform rectangular masking.
  2. CENTROID: A segmentation-free method that uses image intensity to find the vertical center of the retinal band and places rectangular masks there.
  3. ENVELOPE: Uses a pre-trained segmentation model to define a "retinal envelope," ensuring masks overlap with actual tissue.
  4. ANATOMY (V1/V2): Instead of rectangles, the masks are shaped like specific anatomical components (e.g., the inner retina).
  5. COVER: A greedy strategy designed to cover as much anatomy as possible while leaving some visible for context.

The researchers used a ViT-Base encoder and evaluated the learned representations by "linear probing"—freezing the encoder and training a simple linear classifier on top to detect glaucoma.

Figure 1: Top: placement guides. Bottom: target masks. Colours and numbers identify target groups; overlaps are allowed. Illustrative B-scan adapted from Kulyabin et al., OCTDL (arXiv:2312.08255v3), under CC BY 4.0; overlays added.

Datasets

The study utilized the FairVision dataset, which contains OCT volumes labeled for glaucoma.

  • Training/Validation: 6,000 and 1,000 volumes respectively.
  • Test: 3,000 volumes.
  • Each volume contributed 100 B-scans, resized to 256x256 pixels.

Key Insights and Ideas

  • Anatomy Matters: Guiding the model to predict retinal tissue consistently outperformed random masking. By focusing on the "signal" rather than the "noise" of the background, the model learns more clinically relevant features.
  • The "Simplicity" Surprise: The CENTROID method—which requires no manual annotations or complex segmentation models—performed remarkably well. It achieved gains nearly identical to the much more complex ENVELOPE method. This is a win for efficiency in medical AI.
  • Rectangles > Complex Shapes: Interestingly, using masks shaped like actual tissue (ANATOMY-V2) did not improve performance over simple rectangles. The authors suggest that rectangular targets might provide a more challenging and useful learning signal, or that the specific "tissue-shaped" implementation removed too much vital context.
  • The Context Trade-off: Choosing a target is a double-edged sword. While you want the model to predict something meaningful, you also remove that information from the "context" the model uses to make the prediction. Finding the right balance is key.

Key Results

The primary metric was the Area Under the Curve (AUC) for glaucoma classification.

  • ENVELOPE achieved the highest gain at epoch 50, improving over the RANDOM baseline by +1.20 percentage points (pp).
  • CENTROID followed closely with a +0.99 pp improvement at epoch 50, and actually showed the most sustained improvement by epoch 100 (+1.09 pp).
  • More complex strategies like ANATOMY-V2 and COVER were inconsistent and often performed worse than the simple baseline as training progressed.
Performance curves showing that anatomy-guided rectangular targets (CENTROID and ENVELOPE) consistently outperform the random baseline across training epochs.

Limitations and Future Work

  • 2D vs. 3D: The current model processes B-scans (2D slices) independently. Retinal anatomy is inherently 3D, and future work could involve 3D OCT models or "V-JEPA" style temporal/spatial prediction across slices.
  • Single Task: The evaluation was limited to glaucoma classification. It remains to be seen if these anatomy-guided representations translate as well to other retinal diseases (like macular degeneration).
  • Fixed Budgets: The study noted that different strategies naturally result in different amounts of visible context. Future research needs to more strictly isolate the effect of where you mask from how much you mask.

In conclusion, the paper demonstrates that "knowing where to look" is vital for self-supervised medical imaging, and sometimes, a simple intensity-based hint is all the guidance a model needs.

Summary updated

Abstract

Self-supervised learning aims to learn meaningful representations from unlabeled data, potentially allowing models to capture subtle anatomical variations across patients and reduce the amount of labeled data needed. Image-based Joint-Embedding Predictive Architecture (I-JEPA) learns by predicting target-region representations from visible context rather than reconstructing pixels. However, its uniform target placement does not account for the anatomical structure of retinal OCT, where retinal tissue occupies only part of each B-scan. We investigate anatomy-guided target selection using image intensity and predicted retinal segmentation while retaining the I-JEPA architecture and latent prediction objective. We compare anatomy-guided rectangular targets, anatomy-shaped targets and coverage-constrained placement with uniform masking, and evaluate the learned representations using linear probing for glaucoma classification on FairVision. We found that rectangular targets guided by image intensity achieved the highest linear-probe AUC in our comparisons. We show that anatomically meaningful target selection is promising for retinal OCT representation learning, while greater anatomical specificity did not consistently improve performance.
BibTeX
@misc{feng2026where,
  author = {Yunchu Feng and Wai Tak Lau and Kaveri A. Thakoor},
  title = {{Where to Predict in Retinal OCT? Anatomy-Guided Target Selection for I-JEPA}},
  year = {2026},
  eprint = {2611.17586},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17586},
}
Figure