Retinal IPA: Iterative KeyPoints Alignment for Multimodal Retinal Imaging

Jiacheng Wang, Hao Li, Dewei Hu, Rui Xu, Xing Yao, Yuankai K. Tao, Ipek Oguz

Introduction

In modern ophthalmology, comprehensive diagnosis and treatment planning rely on a variety of imaging modalities, including Color Fundus (CF) photography, Fluorescein Angiography (FA), and Optical Coherence Tomography Angiography (OCT-A). Each modality provides unique insights into the retinal structure and vascular health. However, to synthesize these insights into a unified clinical "map," clinicians require precise image registration.

Aligning these images is a significant technical challenge. Multimodal retinal images exhibit substantial "domain shifts"—differences in contrast, texture, and structural representation. Traditional computer vision techniques, such as SIFT or SURF, frequently fail in this context because they are prone to noise from image borders or pathological features like hemorrhages, rather than focusing on stable anatomical landmarks. While deep learning offers a potential solution, most existing models require vast amounts of expert-labeled data, which is prohibitively expensive to obtain in a medical context.

To address these hurdles, we present Retinal IPA (Iterative KeyPoints Alignment), a deep learning framework designed to achieve high-precision multimodal registration using iterative learning and self-supervised vessel segmentation.

Methodology

Retinal IPA builds upon the SuperRetina baseline, introducing three architectural innovations designed to enhance robustness and accuracy in data-scarce environments.

1. Multi-tasking Segmentation

Anatomically, the most reliable keypoints for retinal alignment are located on the vascular tree. Retinal IPA incorporates a segmentation branch that performs a self-supervised vessel segmentation task alongside keypoint detection. By utilizing a consistency loss to predict vessel locations, the model is forced to prioritize domain-agnostic features. This ensures the network ignores transient noise and focuses on the underlying anatomical structures that remain consistent across different imaging modalities.

2. Keypoint-Augmented (KA) SSL Layer

While standard Convolutional Neural Networks (CNNs) excel at capturing local textures, they often lack the global context necessary to distinguish between similar-looking vessel intersections. Retinal IPA introduces a Keypoint-Augmented (KA) layer, which utilizes a transformer-like self-attention mechanism. This layer fuses local CNN-derived features with global geometric context at specific keypoint locations, resulting in feature descriptors that are significantly more robust to local ambiguities.

3. Iterative Training and Bootstrapping

To overcome the lack of large-scale labeled datasets, Retinal IPA employs an iterative training strategy. The framework uses its own predicted keypoints from an initial training phase to refine and "bootstrap" subsequent iterations. This self-correcting loop allows the model to progressively improve its accuracy and reliability, even when starting from a relatively sparse set of initial human-labeled landmarks.

Experimental Evaluation

The framework was evaluated using a diverse range of public and private datasets to test its versatility across species and modalities:

  • Training Data: MeDAL-Retina (labeled and unlabeled fundus images), OCT-500 (2D projections), and an in-house mouse OCT-SLO dataset.
  • Testing Data: The FIRE dataset (single-modality fundus), the CF-FA dataset (multi-modality), and an in-house human OCT-SLO dataset.

Performance Metrics

Retinal IPA consistently outperformed both traditional methods and state-of-the-art deep learning models like SuperPoint and GLAMPoints.

  • Multi-modality Accuracy: On the CF-FA dataset, Retinal IPA achieved a mean Median Error (mMEE) of 2.01, a substantial improvement over SuperRetina’s 3.86.
  • Robustness: On the FIRE dataset, the model achieved an Area Under the Curve (AUC) of 0.778, the highest recorded among tested detector-based methods.
  • Qualitative Results: Visual analysis confirms that Retinal IPA eliminates the "ghosting" or shadowing effects common in other models, producing clean and perfectly overlaid registrations across disparate modalities.

Discussion and Conclusion

The success of Retinal IPA highlights the value of incorporating anatomical priors—such as vessel structure—into the learning process. By treating segmentation as a complementary "side quest," the model learns to focus on stable landmarks. Furthermore, the integration of global context via the KA layer addresses the inherent geometric complexity of the retinal vasculature.

Despite these successes, some limitations remain. The model still benefits from an initial set of sparse human labels to begin the iterative process, and the addition of transformer-like layers increases computational complexity. Future work will focus on optimizing inference speeds for real-time clinical applications and further reducing the reliance on initial human intervention.

Retinal IPA represents a significant step forward in automated retinal image analysis, providing a robust tool for creating the high-fidelity mosaics necessary for advanced ophthalmic care. The code and model weights are available to the research community at https://github.com/MedICL-VU/RetinaIPA.

Summary updated

BibTeX
@inproceedings{wang2024retinal,
  author = {Jiacheng Wang and Hao Li and Dewei Hu and Rui Xu and Xing Yao and Yuankai K. Tao and Ipek Oguz},
  title = {{Retinal IPA: Iterative KeyPoints Alignment for Multimodal Retinal Imaging}},
  booktitle = {MOVI @ MICCAI},
  year = {2024},
  doi = {10.1007/978-3-031-77786-8\_12},
}
Figure