EndoMapper dataset of complete calibrated endoscopy procedures

Pablo Azagra, Carlos Sostres, Ángel Ferrandez, Luis Riazuelo, Clara Tomasini, Oscar León Barbed, Javier Morlana, David Recasens, Victor M. Batlle, Juan J. Gómez Rodríguez, Richard Elvira, Julia López, Cristina Oriol, Javier Civera, Juan D. Tardós, Ana Cristina Murillo, Angel Lanas, José M. M. Montiel

Abstract

Computer-assisted systems are increasingly prevalent in modern medicine. While most endoscopic research focuses on automated pathology detection (e.g., polyps), the localization and navigation of the endoscope remain manual tasks. To advance spatial Artificial Intelligence in this domain, data from complete procedures are essential. We introduce the EndoMapper dataset, the first collection of complete endoscopy sequences acquired during routine clinical practice. Comprising over 15 hours of video across 59 sequences (expanded to 24 hours in later releases), it is the first dataset to include both geometric and photometric endoscope calibration. This resource aims to facilitate the development of Visual Simultaneous Localization and Mapping (VSLAM) for the challenging, non-rigid environment of the gastrointestinal tract.

Introduction: The Challenge of Spatial AI in Endoscopy

While AI has achieved significant success in identifying polyps, the "driving" of the endoscope remains an expert manual task. Enabling advanced features such as augmented reality (AR) overlays, "blind zone" detection, or autonomous robotic navigation requires Spatial AI—the ability of a system to map an organ and track its location within it.

The gastrointestinal (GI) tract presents a unique challenge for standard VSLAM algorithms. Unlike static environments, the colon is a wet, shiny, and highly deformable structure. Furthermore, the light source is co-located with the camera, causing shadows and reflections to shift constantly. Existing datasets typically provide short snippets or focus exclusively on pathology, leaving a gap for researchers tackling the fundamental problems of localization and mapping.

The EndoMapper Dataset

The EndoMapper dataset addresses these gaps by providing complete procedures recorded with high-definition equipment (1440×1080 resolution).

Overview of the EndoMapper dataset, including videos, calibration data, anatomical regions, tool segmentation, and 3D reconstructions.

The dataset includes:

  • Complete Video Sequences: 96 videos totaling approximately 24 hours of footage.
  • Rigorous Calibration: Intrinsic parameters for 10 colonoscopes and 8 gastroscopes using the Kannala-Brandt model to account for fisheye distortion.
  • Rich Annotations: Anatomical landmark labels (e.g., Cecum, Hepatic Flexure), tool segmentation masks, and COLMAP-based 3D reconstructions.
  • Longitudinal Data: Procedures from the same patient taken weeks apart, enabling tests for location recognition across different clinical sessions.

Methodology: Calibration and Simulation

A core contribution of this work is the focus on calibration, which is often overlooked in medical imaging datasets.

Geometric and photometric calibration patterns used to model lens distortion and light fall-off.
  1. Geometric Calibration: By providing precise intrinsic parameters, the dataset allows algorithms to utilize the entire image field, including distorted edges, improving tracking robustness.
  2. Photometric Calibration: The team modeled the endoscope’s light as a "Spot Light Source" (SLS), allowing for mathematical compensation of vignetting and distance-based light attenuation.
  3. Simulated Ground Truth: To overcome the impossibility of obtaining perfect 3D ground truth in vivo, the researchers utilized the VR-Caps simulator to generate synthetic sequences with known geometry and controlled deformations.

Key Insights and Benchmarking

The researchers evaluated the dataset using state-of-the-art tools like ORB-SLAM3 and COLMAP. Several critical observations emerged:

  • The "Rigidity" Trap: Standard VSLAM assumes a rigid world. In endoscopy, the colon deforms, which current systems often misinterpret as camera motion. This led to map fragmentation, with ORB-SLAM3 generating 133 small "sub-maps" for a single sequence rather than one cohesive map.
  • Localization Difficulty: ORB-SLAM3 achieved real-time performance but successfully localized the camera in only about 25% of frames.
  • Anatomical Recognition: Identifying specific sections of the colon remains difficult even for modern CNNs. A ResNetV2-101 model achieved a Top-1 accuracy of only 26.08%, highlighting the complexity of the "where am I?" problem in the GI tract.

Conclusion and Future Directions

EndoMapper serves as a foundational resource for moving beyond simple detection toward comprehensive spatial understanding in endoscopy. The primary limitation remains the lack of 3D ground truth for real human sequences. Future work will likely focus on Non-Rigid VSLAM—algorithms capable of distinguishing between endoscope movement and organ deformation. By capturing the "messy reality" of clinical practice—including bubbles, fluids, and occlusions—EndoMapper provides the necessary training ground for the next generation of medical spatial AI.

Summary updated

Abstract

Computer-assisted systems are becoming broadly used in medicine. In endoscopy, most research focuses on automatic detection of polyps or other pathologies, but localization and navigation of the endoscope is completely performed manually by physicians. To broaden this research and bring spatial Artificial Intelligence to endoscopies, data from complete procedures are needed. This data will be used to build a 3D mapping and localization systems that can perform special task like, for example, detect blind zones during exploration, provide automatic polyp measurements, guide doctors to a polyp found in a previous exploration and retrieve previous images of the same area aligning them for easy comparison. These systems will provide an improvement in the quality and precision of the procedures while lowering the burden on the physicians. This paper introduces the Endomapper dataset, the first collection of complete endoscopy sequences acquired during regular medical practice, including slow and careful screening explorations, making secondary use of medical data. Its original purpose is to facilitate the development and evaluation of VSLAM (Visual Simultaneous Localization and Mapping) methods in real endoscopy data. The first release of the dataset is composed of 59 sequences with more than 15 hours of video. It is also the first endoscopic dataset that includes both the computed geometric and photometric endoscope calibration with the original calibration videos. Meta-data and annotations associated to the dataset varies from anatomical landmark and description of the procedure labeling, tools segmentation masks, COLMAP 3D reconstructions, simulated sequences with groundtruth and meta-data related to special cases, such as sequences from the same patient. This information will improve the research in endoscopic VSLAM, as well as other research lines, and create new research lines.
BibTeX
@inproceedings{azagra2023endomapper,
  author = {Pablo Azagra and Carlos Sostres and Ángel Ferrandez and Luis Riazuelo and Clara Tomasini and Oscar León Barbed and Javier Morlana and David Recasens and Victor M. Batlle and Juan J. Gómez Rodríguez and Richard Elvira and Julia López and Cristina Oriol and Javier Civera and Juan D. Tardós and Ana Cristina Murillo and Angel Lanas and José M. M. Montiel},
  title = {{EndoMapper dataset of complete calibrated endoscopy procedures}},
  booktitle = {Scientific},
  year = {2023},
  doi = {10.1038/s41597-023-02564-7},
}
Figure