Galar - a large multi-label video capsule endoscopy dataset

Maxime Le Floch, Fabian Wolf, Lucian McIntyre, Christoph Weinert, Albrecht Palm, Konrad Volk, P. Herzog, Sophie Helene Kirk, Jonas L. Steinhäuser, C Stopp, Mark Enrik Geissler, Moritz Herzog, Stefan Sulk, Jakob Nikolas Kather, Alexander Meining, Alexander Hann, Stephen P. Palm, Jochen Hampe, Nora Herzog, Franz BrinkmannmedRxiv 2024 · DOI

Abstract

Video capsule endoscopy (VCE) is an important technology with many advantages (non-invasive, representation of small bowel), but faces many limitations as well (time-consuming analysis, short battery lifetime, and poor image quality). Artificial intelligence (AI) holds potential to address every one of these challenges, however the progression of machine learning methods is limited by the availability of extensive data. We propose Galar, the most comprehensive dataset of VCE to date. Galar consists of 80 videos, culminating in 3,513,539 annotated frames covering functional, anatomical, and pathological aspects and introducing a selection of 29 distinct labels. The multisystem and multicenter VCE data from two centers in Saxony (Germany), was annotated framewise and cross-validated by five annotators. The vast scope of annotation and size of Galar make the dataset a valuable resource for the use of AI models in VCE, thereby facilitating research in diagnostic methods, patient care workflow, and the development of predictive analytics in the field.

Introduction

Video Capsule Endoscopy (VCE) is a marvel of non-invasive diagnostics: a patient swallows a pill-sized camera that records their entire digestive tract. However, the procedure generates a "needle in a haystack" problem. Medical staff must often watch hours of footage to find a few seconds of relevant pathology. While Artificial Intelligence (AI) is the obvious solution for automating this review, the field has been hamstrung by a lack of diverse, high-quality data. Existing VCE datasets are typically small, limited to specific pathologies, or sourced from a single hardware system.

To bridge this gap, we introduce Galar, a massive multi-center benchmark designed to provide a robust foundation for training AI to automate the grueling process of manual VCE review.

Dataset Composition and Comparison

The primary contribution of this work is the Galar dataset itself. It comprises over 3.5 million frames sourced from 80 videos across two different medical centers in Germany. To ensure hardware agnosticism, the data includes footage from both Olympus™ and various PillCam™ models.

Comparison of Galar with existing VCE datasets highlights its unique scale and label density.

As shown in the comparison above, while some datasets like Rhode Island contain more raw images, Galar provides a significantly higher density of labels (29 distinct classes). These labels are categorized into technical quality (e.g., bubbles, dirt), anatomical landmarks, and 19 distinct pathological findings, allowing models to understand medical context rather than just identifying disease.

Methodology

The authors developed Galar using a rigorous, multi-step pipeline:

  1. Data Sourcing: Collection of 80 videos from two centers using distinct hardware systems to address the "domain shift" problem.
  2. Annotation: Using the open-source CVAT tool, a team of five annotators labeled every single frame. Frames marked as "unrecognizable" were reviewed by a senior expert with 10 years of experience.
  3. Multi-Label Schema: Implementation of a system covering technical, anatomical, and pathological aspects.
  4. Technical Validation: Training baseline ResNet-50 models to perform multi-label and multi-class classification tasks to prove the dataset's utility.

Key Results

The baseline ResNet-50 models demonstrated that Galar is a high-quality training resource, particularly for anatomical and technical classification.

Baseline performance metrics for anatomical landmark classification.

As detailed in the results, the models performed exceptionally well on large anatomical structures, achieving 93% accuracy for the stomach and 92% for the small intestine. Technical quality classification for "dirt" and "bubbles" reached accuracies of 88% and 82%, respectively. However, detecting rare pathologies remains a challenge; while the AUROC for "blood" was a respectable 0.87, the F1-score was only 0.14, reflecting the massive class imbalance inherent in clinical VCE data.

Limitations and Future Work

While Galar is the most comprehensive dataset to date, it still faces the "long tail" problem common in medical imaging. Certain rare pathologies did not occur in this specific set of 80 videos. Future work will focus on imbalanced learning and more complex loss functions to help AI catch these critical but rare findings. Released under a CC BY 4.0 license, Galar invites the global community to apply advanced architectures like Transformers to improve upon these baseline results.

Summary updated

Abstract

ABSTRACT Video capsule endoscopy (VCE) is an important technology with many advantages (non-invasive, representation of small bowel), but faces many limitations as well (time-consuming analysis, short battery lifetime, and poor image quality). Artificial intelligence (AI) holds potential to address every one of these challenges, however the progression of machine learning methods is limited by the avaibility of extensive data. We propose Galar , the most comprehensive dataset of VCE to date. Galar consists of 80 videos, culminating in 3,513,539 annotated frames covering functional, anatomical, and pathological aspects and introducing a selection of 29 distinct labels. The multisystem and multicenter VCE data from two centers in Saxony (Germany), was annotated framewise and cross-validated by five annotators. The vast scope of annotation and size of Galar make the dataset a valuable resource for the use of AI models in VCE, thereby facilitating research in diagnostic methods, patient care workflow, and the development of predictive analytics in the field.
BibTeX
@misc{floch2024i,
  author = {Maxime Le Floch and Fabian Wolf and Lucian McIntyre and Christoph Weinert and Albrecht Palm and Konrad Volk and P. Herzog and Sophie Helene Kirk and Jonas L. Steinhäuser and C Stopp and Mark Enrik Geissler and Moritz Herzog and Stefan Sulk and Jakob Nikolas Kather and Alexander Meining and Alexander Hann and Stephen P. Palm and Jochen Hampe and Nora Herzog and Franz Brinkmann},
  title = {{<i>Galar</i> - a large multi-label video capsule endoscopy dataset}},
  year = {2024},
  doi = {10.1101/2024.09.26.24314265},
  howpublished = {medRxiv},
}
Figure