FoldIt: Haustral Folds Detection and Segmentation in Colonoscopy Videos

Shawn Mathew, Saad Nadeem, Arie E. Kaufman

Abstract

Colonoscopy is the gold standard for colorectal cancer prevention, yet its effectiveness is limited by "blind spots" created by haustral folds, where polyps frequently hide. We present FoldIt, a semi-supervised generative adversarial network (GAN) designed to detect and segment these folds in real-time optical colonoscopy (OC) videos. By introducing a "transitive loss" that anchors real OC frames to virtual 3D models (VC) and fold annotations, FoldIt achieves superior feature consistency and accuracy. Our model significantly outperforms previous state-of-the-art methods in Dice scores and temporal stability, offering a promising tool for improving surgical navigation and polyp detection.

Introduction

The detection of haustral folds—the natural, accordion-like protrusions of the colon wall—is critical for ensuring complete mucosal coverage during colonoscopy. However, segmenting these folds in real-time is a significant challenge due to the inconsistent lighting, specular reflections, and erratic camera motion characteristic of the procedure. While virtual colonoscopies (VC) derived from CT scans provide a clean 3D map, they often fail to align with live video because the colon distorts during the actual procedure. There is a pressing need for reliable, feature-consistent landmarks to bridge the gap between pre-operative scans and intra-operative video.

Methods: The FoldIt Architecture

FoldIt addresses the domain gap by performing image-to-image translation across three distinct domains:

  1. Domain A: Real Optical Colonoscopy (OC) frames.
  2. Domain B: Haustral fold annotations (red overlays).
  3. Domain C: Virtual Colonoscopy (VC) renderings.

Transitive Loss

Standard unsupervised GANs often struggle with "many-to-many" mappings, leading to visual hallucinations or flickering. FoldIt introduces a transitive loss ($L_T$) to enforce geometric consistency. This loss acts as a triangulation check: translating an image from OC (A) to an annotation (B) and then to a virtual rendering (C) should yield the same result as translating directly from OC (A) to the virtual rendering (C). This constraint forces the network to preserve the underlying topology of the haustral folds across different visual styles.

FoldIt architecture and transitive loss diagram.

The system utilizes four generators and discriminators, optimized through a combination of adversarial, transitive, identity, and ground-truth losses.

Experimental Setup

The study utilized data from 10 patients who underwent both CT-based virtual colonoscopy and traditional optical colonoscopy.

  • Data Volume: 3,000 frames were extracted and split into 300-frame video segments.
  • Ground Truth: Reliable labels were generated using a "Fiedler vector" representation to segment folds on 3D VC meshes, which were then verified by a clinician.
  • Performance: Inference time is approximately 0.04 seconds per image on an Nvidia RTX 6000, enabling real-time application.

Results

FoldIt was evaluated against XDCycleGAN and other recent methods.

Quantitative Performance

FoldIt achieved significantly higher accuracy across various virtual textures:

  • Dice Scores: 0.47–0.50 (vs. 0.21–0.25 for baselines).
  • IoU (Intersection over Union): 0.31–0.33 (vs. 0.12–0.15 for baselines).

Qualitative Consistency

A key advantage of FoldIt is its temporal stability. While previous models often "dropped" or added folds between neighboring frames—causing distracting flickering—FoldIt maintains stable detections throughout video sequences.

Qualitative comparison of FoldIt against state-of-the-art methods across various video sequences.

The model proved robust even when virtual textures were altered, confirming its reliance on geometric features rather than simple color patterns.

Discussion and Future Work

While FoldIt represents a significant step forward, several challenges remain. The model was trained on "clean" virtual images and can struggle with real-world OC frames containing heavy blur or fluid debris. Furthermore, while more stable than its predecessors, the current version lacks an explicit temporal loss function.

Future research will focus on incorporating additional anatomical landmarks, such as the taenia coli (longitudinal muscles), and introducing synthetic noise (blur and fluid) into the training pipeline to enhance real-world robustness. By refining these landmarks, FoldIt aims to provide a reliable "GPS" for clinicians navigating the complex environment of the human colon.

Summary updated

BibTeX
@inproceedings{mathew2021foldit,
  author = {Shawn Mathew and Saad Nadeem and Arie E. Kaufman},
  title = {{FoldIt: Haustral Folds Detection and Segmentation in Colonoscopy Videos}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2021},
  doi = {10.1007/978-3-030-87199-4\_21},
}
Figure