Video Analysis of Small Bowel Capsule Endoscopy Using a Transformer Network

SangYup Oh, Dong Jun Oh, Dongmin Kim, Woohyuk Song, Youngbae Hwang, Nam Ik Cho, Yun Jeong Lim

TL;DR

This paper introduces VWCE-Net, a novel Transformer-based model that analyzes entire small bowel capsule endoscopy videos to detect lesions, overcoming the limitations of traditional methods that rely on manually selected frames. By leveraging the temporal context and global correlations within video sequences, VWCE-Net achieves high sensitivity and specificity, promising to significantly improve automated lesion detection in clinical practice.

What Problem Does This Paper Address?

Wireless Capsule Endoscopy (WCE) is a crucial, non-invasive tool for examining the small bowel, but it generates a massive number of images (over 50,000 per examination). Reviewing these images is incredibly time-consuming for gastroenterologists, often taking over two hours per patient, leading to fatigue and potential missed diagnoses. While deep learning methods have been applied, they typically analyze individual, manually selected "still-shot" images, neglecting the rich temporal information present in the video sequence. This reliance on human selection can also introduce bias and depend on the operator's expertise. The core problem is the need for an automated system that can efficiently and accurately analyze the entire WCE video to detect lesions, capturing the sequential nature of the data.

Methods

The authors propose VWCE-Net, a novel architecture built upon the Transformer model. Unlike previous approaches that treat each image independently, VWCE-Net takes entire video clips as input.

Here's a breakdown of their approach:

  • Video-to-Clip Conversion: The input WCE video is segmented into clips, each consisting of four consecutive frames. This clip size was determined through experimentation to be optimal.
  • Patch Embedding: Each frame within a clip is divided into smaller patches (16x16 pixels). These patches are then flattened and combined with a special clstoken (similar to BERT's approach) to create an embedded feature representation.
  • Positional Encoding: To retain spatial information within the flattened patches, 1D positional encoding is applied.
  • Transformer Architecture: The core of VWCE-Net is a Transformer encoder. This architecture excels at capturing long-range dependencies and global correlations through its self-attention mechanism. It processes the embedded features from the clip, allowing it to learn relationships between different frames and patches.
  • Multi-Head Attention: The model employs multi-head attention, allowing it to focus on different aspects of the input sequence simultaneously and learn complex relationships.
  • Classification: The output from the Transformer layers, specifically the clstoken's representation, is fed into a fully connected layer to classify whether a lesion is present or not. The output dimension is 5, corresponding to the five classes: normal, bleeding, inflammation, vascular, and polyp.

The authors highlight that their method leverages the spatiotemporal self-attention information from sequential images, a departure from traditional image-level analysis.

Datasets

The study utilized a dataset of 260 WCE cases collected at Dongguk University Ilsan Hospital between 2002 and 2022.

  • Training Data: Approximately 1.3 million images from 36 cases were used for training.
  • Test Data: Approximately 170,000 images from 4 cases were used for testing.

The data preparation involved manual labeling of lesions by two gastroenterologists. Importantly, the dataset was structured into clip units (four sequential images) rather than individual still shots. The authors also note the dataset exhibits an imbalance between normal and abnormal images.

Key Insights and Ideas

  • Temporal Context is Crucial: The paper strongly advocates for analyzing WCE videos as sequences rather than isolated frames. The temporal continuity between frames provides vital information that can help resolve ambiguities and reduce false negatives, mimicking how human experts review these videos.
  • Transformer's Strength for Sequential Data: The Transformer architecture, with its self-attention mechanism, is well-suited for processing sequential data like WCE videos. It can effectively capture long-range dependencies and global context, which are essential for understanding the flow of information in a video.
  • Clip Length Matters: The choice of clip length (number of frames per clip) significantly impacts performance. The study found that a clip length of 4 offered the best balance between sensitivity and specificity, suggesting that there's an optimal window for capturing relevant temporal information without introducing too much redundancy or noise.
  • Center Sampling Outperforms Random Sampling: When sampling clips from sequences where lesion presence changes, the "center sampling" method (taking frames from the middle of a segment) yielded better results than random sampling. This implies that the central frames within a continuous segment are more representative of the overall condition.
  • Bridging the Gap to Clinical Practice: By proposing a method that analyzes entire videos, the authors aim to create a system that more closely resembles how human experts diagnose, potentially leading to more efficient and accurate clinical tools.

Key Results

The proposed VWCE-Net model demonstrated strong performance:

  • Sensitivity: 95.1%
  • Specificity: 83.4%

These results were achieved on a test dataset comprising four WCE videos. When compared to existing image-based methods:

  • XceptionNet (classification): Achieved 43.2% sensitivity and 90.4% specificity.
  • YOLOv4 (object detection): Achieved 88.6% sensitivity and 51.5% specificity.

The paper concludes that their video-based approach significantly outperforms image-based methods in terms of overall accuracy.

Limitations and Future Work

  • Dataset Imbalance: The dataset has a significantly larger number of normal images than abnormal ones. This imbalance can negatively impact model performance and increase the risk of overfitting.
  • Redundant Information: Sequential images can be highly similar, leading to redundant information that might hinder effective learning and increase overfitting risk.
  • Noise in Sequences: The authors acknowledge that noise in consecutive images can reduce learning efficiency and degrade performance, suggesting the need for automated preprocessing techniques to remove noise.
  • Future Work: The paper implies that further research into automated preprocessing techniques for sequence data is necessary. Additionally, the authors suggest that developing models that handle consecutive image sequences more effectively will be crucial for advancing lesion detection in WCE. The ultimate goal is to create practical tools that assist gastroenterologists, reduce fatigue, and improve patient care.

Summary updated

Abstract

Although wireless capsule endoscopy (WCE) detects small bowel diseases effectively, it has some limitations. For example, the reading process can be time consuming due to the numerous images generated per case and the lesion detection accuracy may rely on the operators' skills and experiences. Hence, many researchers have recently developed deep-learning-based methods to address these limitations. However, they tend to select only a portion of the images from a given WCE video and analyze each image individually. In this study, we note that more information can be extracted from the unused frames and the temporal relations of sequential frames. Specifically, to increase the accuracy of lesion detection without depending on experts' frame selection skills, we suggest using whole video frames as the input to the deep learning system. Thus, we propose a new Transformer-architecture-based neural encoder that takes the entire video as the input, exploiting the power of the Transformer architecture to extract long-term global correlation within and between the input frames. Subsequently, we can capture the temporal context of the input frames and the attentional features within a frame. Tests on benchmark datasets of four WCE videos showed 95.1% sensitivity and 83.4% specificity. These results may significantly advance automated lesion detection techniques for WCE images.
BibTeX
@inproceedings{oh2023video,
  author = {SangYup Oh and Dong Jun Oh and Dongmin Kim and Woohyuk Song and Youngbae Hwang and Nam Ik Cho and Yun Jeong Lim},
  title = {{Video Analysis of Small Bowel Capsule Endoscopy Using a Transformer Network}},
  booktitle = {Diagnostics},
  year = {2023},
  doi = {10.3390/diagnostics13193133},
}
Figure