Deep Transformers for Fast Small Intestine Grounding in Capsule Endoscope Video

Xinkai Zhao, Chaowei Fang, Feng Gao, De-Jun Fan, Xutao Lin, Guanbin Li

Abstract

Capsule endoscopy is an evolutional technique for examining and diagnosing intractable gastrointestinal diseases. Because of the huge amount of data, analyzing capsule endoscope videos is very time-consuming and labor-intensive for gastrointestinal medicalists. The development of intelligent long video analysis algorithms for regional positioning and analysis of capsule endoscopic video is therefore essential to reduce the workload of clinicians and assist in improving the accuracy of disease diagnosis. In this paper, we propose a deep model to ground shooting range of small intestine from a capsule endoscope video which has duration of tens of hours. This is the first attempt to attack the small intestine grounding task using deep neural network method. We model the task as a 3-way classification problem, in which every video frame is categorized into esophagus/stomach, small intestine or colorectum. To explore long-range temporal dependency, a transformer module is built to fuse features of multiple neighboring frames. Based on the classification model, we devise an efficient search algorithm to efficiently locate the starting and ending shooting boundaries of the small intestine. Without searching the small intestine exhaustively in the full video, our method is implemented via iteratively separating the video segment along the direction to the target boundary in the middle. We collect 113 videos from a local hospital to validate our method. In the 5-fold cross validation, the average IoU between the small intestine segments located by our method and the ground-truths annotated by broad-certificated gastroenterologists reaches 0.945.

Introduction

Capsule endoscopy (CE) has revolutionized non-invasive imaging of the gastrointestinal (GI) tract. However, the procedure generates a massive data bottleneck: a single video can last over 10 hours and contain more than 100,000 frames. A critical clinical prerequisite for diagnosis is "grounding"—identifying the specific temporal segment where the capsule is within the small intestine. Manually locating the pylorus (start) and ileocecal valve (end) is a labor-intensive task that delays actual diagnostic work.

While previous AI research has focused on lesion detection, this study addresses the foundational problem of regional positioning. By automatically cropping the video to the relevant small intestine segment, clinicians can focus their expertise where it is most needed, significantly reducing the workload.

Methodology

The researchers proposed a three-stage deep learning pipeline to automate the grounding process, combining visual feature extraction with temporal context modeling.

The three-stage deep learning framework for small intestine grounding.

Stage 1: Feature Extraction

Every video frame is passed through a CNN encoder (such as ResNet) to extract high-level visual features. These features capture the mucosal textures and anatomical landmarks unique to different sections of the GI tract.

Stage 2: Temporal Feature Enhancement (TFE)

Because the sequence of organs is physiologically fixed (stomach $\rightarrow$ small intestine $\rightarrow$ colon), the identity of a frame is highly dependent on its neighbors. The framework utilizes a Transformer module to fuse features from a sliding window of frames. This self-attention mechanism captures long-range temporal dependencies more effectively than standard LSTMs, which is crucial for identifying transition points.

Stage 3: Efficient Grounding Module

To avoid the computational cost of exhaustive classification, the authors developed a "fault-tolerant" iterative search algorithm. Starting from the video's midpoint, the algorithm uses the model's classification confidence to determine adaptive step sizes. It "slows down" near suspected boundaries, allowing for high-precision localization in an average of just 109 seconds per full-length video.

Results and Discussion

The model was validated using 113 CE videos from the Sixth Affiliated Hospital, Sun Yat-sen University. The ResNet + Transformer (TFE) architecture achieved an average Intersection over Union (IoU) of 0.945, significantly outperforming baseline architectures like VGG (0.601) or MobileNet (0.800).

Error Analysis

The precision of the grounding algorithm was assessed by measuring the frame-distance error between predicted and expert-annotated boundaries.

Error analysis for start and end frame localization across different models.

As shown in the error analysis, the Transformer-based approach (+TFE) consistently yields the lowest median error and the tightest distribution compared to standard ResNet or LSTM-augmented versions. For most samples, the error between the predicted boundary and the clinician's annotation was within 100 frames—a negligible margin in a 100,000-frame video.

Conclusion

This work introduces the first deep learning framework for small intestine grounding in capsule endoscopy. By leveraging Transformers for temporal context and an adaptive search algorithm, the system provides a robust solution to a major clinical bottleneck. Future work will focus on validating the model across multi-center datasets and diverse hardware platforms to ensure broad clinical applicability.

Summary updated

Abstract

Capsule endoscopy is an evolutional technique for examining and diagnosing intractable gastrointestinal diseases. Because of the huge amount of data, analyzing capsule endoscope videos is very time-consuming and labor-intensive for gastrointestinal medicalists. The development of intelligent long video analysis algorithms for regional positioning and analysis of capsule endoscopic video is therefore essential to reduce the workload of clinicians and assist in improving the accuracy of disease diagnosis. In this paper, we propose a deep model to ground shooting range of small intestine from a capsule endoscope video which has duration of tens of hours. This is the first attempt to attack the small intestine grounding task using deep neural network method. We model the task as a 3-way classification problem, in which every video frame is categorized into esophagus/stomach, small intestine or colorectum. To explore long-range temporal dependency, a transformer module is built to fuse features of multiple neighboring frames. Based on the classification model, we devise an efficient search algorithm to efficiently locate the starting and ending shooting boundaries of the small intestine. Without searching the small intestine exhaustively in the full video, our method is implemented via iteratively separating the video segment along the direction to the target boundary in the middle. We collect 113 videos from a local hospital to validate our method. In the 5-fold cross validation, the average IoU between the small intestine segments located by our method and the ground-truths annotated by broad-certificated gastroenterologists reaches 0.945.
BibTeX
@inproceedings{zhao2021deep,
  author = {Xinkai Zhao and Chaowei Fang and Feng Gao and De-Jun Fan and Xutao Lin and Guanbin Li},
  title = {{Deep Transformers for Fast Small Intestine Grounding in Capsule Endoscope Video}},
  booktitle = {ISBI},
  year = {2021},
  url = {https://openreview.net/forum?id=u_Sy1zQYww},
}
Figure