Bilateral-ViT For Robust Fovea Localization

Sifan Song, Kang Dang, Qinji Yu, Zilong Wang, Frans Coenen, Jionglong Su, Xiaowei Ding

TL;DR

The Bilateral-ViT is a novel neural network architecture that achieves state-of-the-art fovea localization in retinal images by combining global context from Vision Transformers with structural cues from blood vessel maps. This "bilateral" approach makes the model exceptionally robust, particularly in diseased eyes where the fovea is often obscured by lesions or hemorrhages.

What Problem Does This Paper Address?

Locating the fovea—the central point of the retina responsible for sharp vision—is a critical step in diagnosing retinal diseases. However, the fovea is notoriously difficult to detect automatically. It often appears as a low-contrast, "fuzzy" region. In patients with conditions like diabetic retinopathy or high myopia, the fovea can be completely hidden by blood, lesions, or atrophic patches.

While traditional Convolutional Neural Networks (CNNs) are great at detecting local textures, they often lack the "global" perspective needed to infer the fovea's location based on the surrounding anatomy when the fovea itself is invisible. This paper addresses the need for a localization method that is robust enough to handle severely diseased retinal images.

Methods

The authors propose Bilateral-ViT, a dual-branch architecture based on a U-shaped design:

  1. Main Branch (The "Context" Branch): This branch uses TransUNet, a hybrid architecture that combines CNN layers with 12 Transformer blocks. The CNN extracts low-level features, while the Transformer blocks use self-attention to model long-range dependencies across the entire fundus image.
  2. Vessel Branch (The "Structural" Branch): Recognizing that blood vessels naturally "point" toward the macula, the authors designed a branch that specifically processes a blood vessel segmentation map (generated by a pre-trained model). This branch uses Spatial Information Guidance (SIG) blocks, which utilize Residual U-blocks (RSU) to capture high-resolution spatial structures.
  3. Multi-scale Feature Fusion (MFF): Instead of simply concatenating the outputs, the model uses customized MFF blocks to merge features from both branches at multiple scales. This ensures that the global context from the Transformer and the structural roadmap from the vessel branch are effectively integrated before the final coordinate prediction.

Datasets

The model was evaluated on two major benchmarks:

  • Messidor: 1,136 images focused on diabetic retinopathy (540 normal, 660 diseased).
  • PALM (Pathologic Myopia Challenge): 400 images characterized by extreme "pathologic myopia," which involves significant structural distortion and large lesions, making fovea localization much harder than in standard datasets.

Key Insights and Ideas

  • Transformers as Global Navigators: In medical imaging, we often think "local." However, for fovea localization, the "global" view is essential. Transformers allow the model to look at the optic disc and the overall vessel arch to "guess" where the fovea should be, even if that area is covered by a hemorrhage.
  • Vessels as a Prior: By explicitly feeding a vessel map into a separate branch, the authors provide the network with a geometric "cheat sheet." This is more effective than asking a single-branch network to learn both the raw pixel intensities and the complex vascular geometry simultaneously.
  • Robustness to Disease: The most striking insight is the performance gap on diseased images. While standard models (like U-Net) see their accuracy plummet when lesions are present, Bilateral-ViT maintains high precision, suggesting it has learned the underlying anatomical "rules" rather than just memorizing local textures.

Key Results

The Bilateral-ViT established new state-of-the-art (SOTA) results across the board:

  • Messidor Dataset: Achieved 100% accuracy at the $1/2 R$ threshold (where $R$ is the optic disc radius), meaning every single fovea was located within a very tight margin.
  • PALM (Diseased Images): On the most challenging diseased images in the PALM dataset, Bilateral-ViT achieved 92.45% accuracy at the $1R$ threshold, outperforming the previous best model (TransUNet) by a massive 17% margin.
  • Cross-Dataset Generalization: When trained on PALM and tested on Messidor, the model achieved 99.38% accuracy, proving it isn't just over-fitting to one specific camera type or patient population.

Limitations and Future Work

While the paper demonstrates impressive results, a few areas remain for exploration:

  • Two-Step Dependency: The vessel branch relies on a pre-trained vessel segmentation model. If the initial vessel segmentation is poor (e.g., in extremely low-quality images), it might negatively impact the fovea localization.
  • Computational Overhead: Transformers and dual-branch architectures are more computationally expensive than simple CNNs. The authors do not deeply discuss the inference latency, which is a factor for real-time clinical applications.
  • Future Directions: The authors suggest that this bilateral framework could be extended to other anatomical landmarks or even joint tasks, such as simultaneously segmenting lesions and locating the fovea.

Summary updated

Abstract

The fovea is an important anatomical landmark of the retina. Detecting the location of the fovea is essential for the analysis of many retinal diseases. However, robust fovea localization remains a challenging problem, as the fovea region often appears fuzzy, and retina diseases may further obscure its appearance. This paper proposes a novel Vision Transformer (ViT) approach that integrates information both inside and outside the fovea region to achieve robust fovea localization. Our proposed network, named Bilateral-Vision-Transformer (Bilateral-ViT), consists of two network branches: a transformer-based main network branch for integrating global context across the entire fundus image and a vessel branch for explicitly incorporating the structure of blood vessels. The encoded features from both network branches are subsequently merged with a customized Multi-scale Feature Fusion (MFF) module. Our comprehensive experiments demonstrate that the proposed approach is significantly more robust for diseased images and establishes the new state of the arts using the Messidor and PALM datasets.
BibTeX
@inproceedings{song2022bilateral,
  author = {Sifan Song and Kang Dang and Qinji Yu and Zilong Wang and Frans Coenen and Jionglong Su and Xiaowei Ding},
  title = {{Bilateral-ViT For Robust Fovea Localization}},
  booktitle = {ISBI},
  year = {2022},
  doi = {10.1109/isbi52829.2022.9761523},
}
Figure