Novel OCT mosaicking pipeline with Feature- and Pixel-based registration

Jiacheng Wang, Hao Li, Dewei Hu, Yuankai K. Tao, Ipek Oguz

TL;DR

Researchers at Vanderbilt University have developed a robust four-stage pipeline for stitching narrow-field Optical Coherence Tomography (OCT) images into high-resolution panoramas. By bridging deep-learning-based feature matching with classical deformable registration and utilizing the Segment Anything Model (SAM) for unsupervised verification, their method achieves superior accuracy and efficiency even in the presence of significant noise and motion artifacts.

What Problem Does This Paper Address?

Optical Coherence Tomography (OCT) is a cornerstone of modern ophthalmology, providing high-resolution cross-sectional views of the retina. However, it suffers from a "keyhole" problem: to maintain high resolution, the field of view (FoV) is typically very narrow. To see the "big picture" of a patient's eye, clinicians must stitch multiple overlapping images together—a process called mosaicking.

Mosaicking OCT images is notoriously difficult. The images are often noisy, and patient movement between scans creates non-linear deformations. Traditional feature-based methods (like SIFT) often fail in noisy medical scans, while pixel-based methods (which look at raw intensity) have a narrow "capture range," meaning they easily get stuck in local optima if the initial alignment isn't nearly perfect. Furthermore, evaluating these mosaics is hard because there is rarely a "ground truth" wide-field image to compare against.

Methods

The authors propose a coarse-to-fine pipeline divided into four distinct stages, illustrated in the figure below:

Fig. 1. Four-stage OCT mosaicking pipeline. Stage I displays feature matching: the moving image in blue and the reference in purple, and 5 sample matched features. Stage II shows the features-intensity affine transform applied to the moving image. In Stage III, we use deformable registration to refine the alignment. In Stage IV, the final mosaic image is evaluated by SAM.
  1. Coarse Stage (Feature-based): The system uses SuperPoint for feature detection and LightGlue for matching. LightGlue is a state-of-the-art transformer-based matcher that is faster and more accurate than its predecessors.
  2. Bridge Stage (Feature-Intensity Affine): This is a key innovation. Instead of using standard RANSAC-based homography (which can cause shearing distortions), the authors create "feature intensity maps" where matched points are assigned unique intensities. They then perform a global intensity-based affine registration on these maps to find a stable initial alignment.
  3. Refinement Stage (Pixel-based): With a solid initial alignment from the Bridge stage, the pipeline uses the Advanced Normalization Tools (ANTs) SyN algorithm to perform non-rigid, deformable registration. This fine-tunes the alignment at the pixel level to account for local tissue deformations.
  4. Verification Stage (Unsupervised SAM): To solve the evaluation problem, they use the Segment Anything Model (SAM). By using the matched features as "prompts," they segment vascular structures in both the reference and the resulting mosaic. If the mosaicking is poor, "ghost vessels" appear, causing the segmentation to fail or diverge, which can be measured quantitatively.

Datasets

The pipeline was evaluated on two distinct datasets:

  • In-house 9-field dataset: High-resolution OCT sub-fields from two healthy subjects, covering a 45-degree FoV. These images were intentionally not denoised to test the pipeline's robustness.
  • Synthetic OCTA-500 dataset: Based on a public dataset of 300 human OCTA projections. The authors created synthetic sub-fields with known random rotations, translations, and elastic deformations to provide a quantitative benchmark with ground truth.

Key Insights and Ideas

  • LightGlue is the "Goldilocks" Matcher: The authors found that while dense matching methods like LoFTR provide many points, they are computationally expensive and often include less informative features. LightGlue provides a sparse but highly representative set of matches that is ideal for medical imaging.
  • The "Bridge" Strategy: By converting point matches into an intensity map, the authors leverage the stability of global correlation metrics. This prevents the "local optima" trap of pixel-based methods while avoiding the geometric distortions common in pure feature-based homography.
  • SAM as a Quality Auditor: Using a foundation model like SAM for verification is a clever "aha moment." It treats the consistency of biological structures (like blood vessels) as a proxy for registration accuracy, allowing for automated quality control without needing manual labels.

Key Results

The proposed pipeline outperformed standard methods across multiple metrics:

  • Accuracy: On the synthetic dataset, LightGlue achieved a rotation error of only 0.52°, significantly better than SIFT (1.75°) or SuperGlue (1.21°).
  • Robustness: In the in-house dataset, the "Affine + SyN" approach showed significantly better performance in Mean Square Error (MSE), Structural Similarity (SSIM), and Peak Signal-to-Noise Ratio (PSNR) compared to the standard MAGSAC method.
  • Qualitative Improvement: As shown in Figure 4, the refinement stage successfully eliminated "ghosting" artifacts in vessel structures that were present in coarser methods. The SAM-based Dice score improved from 0.53 (standard MAGSAC) to 0.75 with the full proposed pipeline.
Fig. 4. (A) Quantitative comparison in the 9-field dataset: Light colors (MAGSAC and Affine) represent coarse stage results, while dark colors (MAGSAC+SyN and Affine+SyN) represent the refinement stage. Our method (Affine+SyN) outperformed MAGSAC in all three metrics at both stages. (B) Qualitative comparison, with pink arrows highlighting noteworthy differences. The intensity difference between the mosaic and reference images (∆= |Iref −Imosaic|) in the dashed region is represented using a blue-to-red color map. (C) SAM segmentation from the mosaic is compared against the reference image segmentation. Blue areas indicate segmentation agreement, while orange regions reveal discrepancies. Dice score for each method is reported.

Limitations and Future Work

While the results are promising, the authors note a few areas for growth:

  • Sample Size: The real-world validation was performed on a limited number of subjects (two subjects, though yielding 144 image pairs). Testing on a broader clinical population with various pathologies would be a logical next step.
  • Evaluation Metrics: While the SAM-based Dice score is a strong start for unsupervised evaluation, the authors suggest exploring other metrics like Average Surface Distance (ASD) and Hausdorff distance for even more rigorous verification.

The code and evaluation tools for this project have been made publicly available on GitHub.

Summary updated

Abstract

High-resolution Optical Coherence Tomography (OCT) images are crucial for ophthalmology studies but are limited by their relatively narrow field of view (FoV). Image mosaicking is a technique for aligning multiple overlapping images to obtain a larger FoV. Current mosaicking pipelines often struggle with substantial noise and considerable displacement between the input sub-fields. In this paper, we propose a versatile pipeline for stitching multi-view OCT/OCTA \textit{en face} projection images. Our method combines the strengths of learning-based feature matching and robust pixel-based registration to align multiple images effectively. Furthermore, we advance the application of a trained foundational model, Segment Anything Model (SAM), to validate mosaicking results in an unsupervised manner. The efficacy of our pipeline is validated using an in-house dataset and a large public dataset, where our method shows superior performance in terms of both accuracy and computational efficiency. We also made our evaluation tool for image mosaicking and the corresponding pipeline publicly available at \url{https://github.com/MedICL-VU/OCT-mosaicking}.
BibTeX
@misc{wang2023novel,
  author = {Jiacheng Wang and Hao Li and Dewei Hu and Yuankai K. Tao and Ipek Oguz},
  title = {{Novel OCT mosaicking pipeline with Feature- and Pixel-based registration}},
  year = {2023},
  eprint = {2311.13052},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.13052},
}
Figure