Abstract
Computer-assisted systems are increasingly prevalent in modern medicine. While most endoscopic research focuses on automated pathology detection (e.g., polyps), the localization and navigation of the endoscope remain manual tasks. To advance spatial Artificial Intelligence in this domain, data from complete procedures are essential. We introduce the EndoMapper dataset, the first collection of complete endoscopy sequences acquired during routine clinical practice. Comprising over 15 hours of video across 59 sequences (expanded to 24 hours in later releases), it is the first dataset to include both geometric and photometric endoscope calibration. This resource aims to facilitate the development of Visual Simultaneous Localization and Mapping (VSLAM) for the challenging, non-rigid environment of the gastrointestinal tract.
Introduction: The Challenge of Spatial AI in Endoscopy
While AI has achieved significant success in identifying polyps, the "driving" of the endoscope remains an expert manual task. Enabling advanced features such as augmented reality (AR) overlays, "blind zone" detection, or autonomous robotic navigation requires Spatial AI—the ability of a system to map an organ and track its location within it.
The gastrointestinal (GI) tract presents a unique challenge for standard VSLAM algorithms. Unlike static environments, the colon is a wet, shiny, and highly deformable structure. Furthermore, the light source is co-located with the camera, causing shadows and reflections to shift constantly. Existing datasets typically provide short snippets or focus exclusively on pathology, leaving a gap for researchers tackling the fundamental problems of localization and mapping.
The EndoMapper Dataset
The EndoMapper dataset addresses these gaps by providing complete procedures recorded with high-definition equipment (1440×1080 resolution).
The dataset includes:
- Complete Video Sequences: 96 videos totaling approximately 24 hours of footage.
- Rigorous Calibration: Intrinsic parameters for 10 colonoscopes and 8 gastroscopes using the Kannala-Brandt model to account for fisheye distortion.
- Rich Annotations: Anatomical landmark labels (e.g., Cecum, Hepatic Flexure), tool segmentation masks, and COLMAP-based 3D reconstructions.
- Longitudinal Data: Procedures from the same patient taken weeks apart, enabling tests for location recognition across different clinical sessions.
Methodology: Calibration and Simulation
A core contribution of this work is the focus on calibration, which is often overlooked in medical imaging datasets.
- Geometric Calibration: By providing precise intrinsic parameters, the dataset allows algorithms to utilize the entire image field, including distorted edges, improving tracking robustness.
- Photometric Calibration: The team modeled the endoscope’s light as a "Spot Light Source" (SLS), allowing for mathematical compensation of vignetting and distance-based light attenuation.
- Simulated Ground Truth: To overcome the impossibility of obtaining perfect 3D ground truth in vivo, the researchers utilized the VR-Caps simulator to generate synthetic sequences with known geometry and controlled deformations.
Key Insights and Benchmarking
The researchers evaluated the dataset using state-of-the-art tools like ORB-SLAM3 and COLMAP. Several critical observations emerged:
- The "Rigidity" Trap: Standard VSLAM assumes a rigid world. In endoscopy, the colon deforms, which current systems often misinterpret as camera motion. This led to map fragmentation, with ORB-SLAM3 generating 133 small "sub-maps" for a single sequence rather than one cohesive map.
- Localization Difficulty: ORB-SLAM3 achieved real-time performance but successfully localized the camera in only about 25% of frames.
- Anatomical Recognition: Identifying specific sections of the colon remains difficult even for modern CNNs. A ResNetV2-101 model achieved a Top-1 accuracy of only 26.08%, highlighting the complexity of the "where am I?" problem in the GI tract.
Conclusion and Future Directions
EndoMapper serves as a foundational resource for moving beyond simple detection toward comprehensive spatial understanding in endoscopy. The primary limitation remains the lack of 3D ground truth for real human sequences. Future work will likely focus on Non-Rigid VSLAM—algorithms capable of distinguishing between endoscope movement and organ deformation. By capturing the "messy reality" of clinical practice—including bubbles, fluids, and occlusions—EndoMapper provides the necessary training ground for the next generation of medical spatial AI.
Summary updated