Automatic Image Unfolding and Stitching Framework for Esophageal Lining Video Based on Density-Weighted Feature Matching

Muyang Li, Juming Xiong, Ruining Deng, Tianyuan Yao, Regina N Tyree, Girish Hiremath, Yuankai Huo

TL;DR

This paper introduces an automated framework that transforms narrow, tubular endoscopic videos of the esophagus into flat, continuous panoramic views. By combining deep-learning depth estimation with a novel density-weighted feature matching algorithm, the method significantly improves stitching accuracy and structural consistency over traditional approaches, paving the way for better clinical gastrointestinal assessments.

What Problem Does This Paper Address?

Endoscopy is a vital tool for diagnosing gastrointestinal tract disorders, but doctors are often forced to work with a very narrow field of view (FOV). This limitation is especially problematic in the esophagus, where the camera's close proximity and magnification obscure the broader context of the tissue.

While image stitching can theoretically combine multiple video frames into a wide panorama, the esophagus presents a nightmare scenario for standard computer vision algorithms. It is a dynamic, constantly moving tube filled with intricate, highly repetitive patterns that lack the distinct landmarks found in natural images. Traditional stitching methods frequently suffer from severe warping, misalignment, and distortion under these conditions.

Methods

The authors tackle this challenge with a three-step pipeline that bridges deep learning and classic geometric computer vision:

  1. Image Unfolding: To flatten the tubular structure of the esophagus, the framework uses a state-of-the-art monocular depth estimation model called Depth Anything. By identifying the pixel with the minimum depth (the center of the esophageal lumen), the system extracts an annular (donut-shaped) region of interest and unwraps it into a flat, rectangular image using a Virtual Cylindrical Grid (VCG).
  2. Feature Filtering Pool: Instead of relying on a single feature detection algorithm, the framework pools keypoints from both traditional methods (SIFT, ORB) and a modern transformer-based deep learning method (LoFTR). It then filters these points using a threshold based on the depth center offset to ensure a parallel, consistent horizontal distribution across frames.
  3. Density-Weighted Homography Optimization (DWHO): Standard image stitching can be easily skewed if one area of an image has a massive cluster of feature points, causing the algorithm to over-optimize for that spot while misaligning the rest of the image. The authors introduce DWHO, which calculates density weights to prevent feature-dense regions from disproportionately influencing the homography matrix, ensuring a balanced and seamless stitch.
Figure 1: Pipeline for esophageal image unfolding, matching, and stitching. The process includes three main steps: 1) transforming circular video views into unfolded images; 2) pooling and filtering feature points from deep-learning and traditional methods; 3) estimating horizontal and vertical displacements in overlapping regions to create a stitched representation of the esophageal inner surface.

Datasets

The framework was evaluated on a clinical dataset of 5 endoscopy videos collected from 5 pediatric patients (aged 6–18) diagnosed with Eosinophilic Esophagitis (EoE) at the Monroe Carrell Jr. Children’s Hospital at Vanderbilt University Medical Center (VUMC). Quantitative evaluations were performed individually across 300 frames selected from these video sequences.

Key Insights and Ideas

  • Depth as a Stabilization Anchor: Using a depth estimation model to find the "center" of a biological tube provides a remarkably stable coordinate system over time. Tracking this center trajectory allows for highly consistent unwrapping before stitching even begins.
  • The Danger of Feature Clumping: In medical imaging, certain tissue textures generate dense clusters of features. The core "aha moment" of the DWHO algorithm is realizing that unweighted homography optimization allows these clusters to dominate the alignment math, leading to global warping. Weighting by inverse density solves this issue.
  • Hybrid Synergy: Combining the dense, pixel-wise matching capabilities of Transformers (LoFTR) with the localized precision of classic descriptors (SIFT/ORB) creates a more robust feature pool than any single method could achieve alone.

Key Results

The proposed DWHO framework outperformed standalone SIFT, ORB, and LoFTR methods across both quantitative metrics and qualitative visual appeal:

  • Structural Similarity Index (SSIM): DWHO achieved the highest average structural similarity score of 0.502, outperforming LoFTR (0.492), SIFT (0.461), and ORB (0.454).
  • Root Mean Square Error (RMSE): DWHO achieved the lowest alignment error with an average RMSE of 5.776, compared to LoFTR (5.815), SIFT (5.852), and ORB (5.888).

A Wilcoxon signed-rank test confirmed that these improvements were statistically significant ($p < 0.05$). Visually, the final panoramas generated by DWHO showed superior continuity, fewer alignment artifacts, and a much flatter, more readable representation of the esophageal lining.

Figure 3: Quantitative and Qualitative Comparison of Image Matching Methods. Top: Box plots comparing Structural Similarity Index (SSIM) and Root Mean Square Error (RMSE) across different methods (SIFT, ORB, LOFTR, DWHO). Bottom: Visualized stitched results for each method, highlighting the performance in generating coherent and accurate image reconstructions.

Limitations and Future Work

The authors note that deep learning-based stitching methods are still in their infancy regarding medical imagery applications. While the framework shows strong performance on this specific pediatric EoE cohort, broader validation on larger, more diverse datasets featuring different esophageal pathologies (like Barrett's esophagus or motility disorders) will be necessary to prove its clinical versatility. Future work could also explore real-time optimization to allow clinicians to view these unfolded panoramas live during an endoscopic procedure.

Summary updated

Abstract

Endoscopy is a crucial tool for diagnosing the gastrointestinal tract, but its effectiveness is often limited by a narrow field of view and the dynamic nature of the internal environment, especially in the esophagus, where complex and repetitive patterns make image stitching challenging. This paper introduces a novel automatic image unfolding and stitching framework tailored for esophageal videos captured during endoscopy. The method combines feature matching algorithms, including LoFTR, SIFT, and ORB, to create a feature filtering pool and employs a Density-Weighted Homography Optimization (DWHO) algorithm to enhance stitching accuracy. By merging consecutive frames, the framework generates a detailed panoramic view of the esophagus, enabling thorough and accurate visual analysis. Experimental results show the framework achieves low Root Mean Square Error (RMSE) and high Structural Similarity Index (SSIM) across extensive video sequences, demonstrating its potential for clinical use and improving the quality and continuity of endoscopic visual data.
BibTeX
@misc{li2024automatic,
  author = {Muyang Li and Juming Xiong and Ruining Deng and Tianyuan Yao and Regina N Tyree and Girish Hiremath and Yuankai Huo},
  title = {{Automatic Image Unfolding and Stitching Framework for Esophageal Lining Video Based on Density-Weighted Feature Matching}},
  year = {2024},
  eprint = {2410.01148},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.01148},
}
Figure