Photoshopping Colonoscopy Video Frames

Yuyuan Liu, Yu Tian, Gabriel Maicas, Leonardo Z. C. T. Pu, Rajvinder Singh, Johan W. Verjans, Gustavo Carneiro

Abstract: The automatic detection of frames containing polyps from a colonoscopy video sequence is an important first step for a fully automated colonoscopy analysis tool. Typically, such detection system is built using a large annotated data set of frames with and without polyps, which is expensive to be obtained. In this paper, we introduce a new system that detects frames containing polyps as anomalies from a distribution of frames from exams that do not contain any polyps. The system is trained using a one-class training set consisting of colonoscopy frames without polyps -- such training set is considerably less expensive to obtain, compared to the 2-class data set mentioned above. During inference, the system is only able to reconstruct frames without polyps, and when it tries to reconstruct a frame with polyp, it automatically removes (i.e., photoshop) it from the frame -- the difference between the input and reconstructed frames is used to detect frames with polyps. We name our proposed model as anomaly detection generative adversarial network (ADGAN), comprising a dual GAN with two generators and two discriminators. We show that our proposed approach achieves the state-of-the-art result on this data set, compared with recently proposed anomaly detection systems.

Introduction

Colorectal cancer is the third leading cause of cancer-related deaths, yet early detection of polyps during colonoscopies can increase survival rates to over 90%. Manual detection, however, is difficult and prone to human error due to the visual variety of polyps and clinician fatigue.

While deep learning has advanced medical imaging, most existing systems are "2-class classifiers" requiring massive, hand-labeled datasets of both healthy and diseased tissue. These datasets are expensive and time-consuming to produce. Furthermore, polyps vary wildly in appearance, making it difficult to train a model that recognizes every possible variation. This paper addresses these challenges by reframing the task as unsupervised anomaly detection: by learning exactly what a healthy colon looks like, any polyp—regardless of its shape—stands out as an anomaly.

Methodology: The ADGAN Architecture

The researchers introduced the Anomaly Detection Generative Adversarial Network (ADGAN). Unlike standard GANs, ADGAN uses a dual-structure approach to ensure a tight mapping between images and their mathematical "latent" representations. The system consists of four main components:

  1. Visual Generator ($G_v$): Creates a colonoscopy image from a latent vector.
  2. Visual Discriminator ($D_v$): Distinguishes between real healthy images and those created by the generator.
  3. Latent Generator ($G_l$): Maps an image back into a latent vector.
  4. Latent Discriminator ($D_l$): Ensures the produced latent vectors match the expected distribution.

The "Photoshop" Mechanism

Because the model is trained exclusively on healthy frames, it lacks the "vocabulary" to recreate a polyp. During inference, when the model encounters a test frame containing a polyp, it attempts to reconstruct it as healthy tissue. This effectively "photoshops" the anomaly out of the frame. By calculating the reconstruction error—the pixel-wise difference between the input and the output—the system can pinpoint the exact location of the polyp.

Visual demonstration of ADGAN 'photoshoping' polyps out of reconstructed frames.

Experimental Setup

The study utilized a colonoscopy dataset derived from 18 videos across 15 patients:

  • Training Set: 13,350 "normal" (healthy) images.
  • Testing Set: 967 images (290 abnormal frames with polyps and 677 normal frames).
  • Preprocessing: Images were downsampled to 64x64 resolution, and frames containing heavy blur or debris were removed to ensure the model learned from clean tissue.

Key Results

ADGAN was compared against several baselines, including Deep Auto-Encoders (DAE), Variational Auto-Encoders (VAE), and other GAN-based detectors like OC-GAN and f-AnoGAN.

Comparison of AUC scores across different anomaly detection methods.

As shown in the results table, ADGAN achieved a state-of-the-art AUC of 0.7296, significantly outperforming the next best model, f-AnoGAN (0.6913). The dual-constraint setup of ADGAN prevents the model from accidentally learning to reconstruct anomalies, which is a common failure point in standard autoencoders.

Conclusion and Future Work

ADGAN demonstrates that unsupervised learning can effectively identify medical anomalies while reducing the need for expensive labeled data. Future work will focus on handling real-world "messiness" (such as feces or camera blur) without triggering false positives, increasing image resolution, and leveraging temporal video data to improve detection consistency across multiple frames.

Summary updated

Abstract

The automatic detection of frames containing polyps from a colonoscopy video sequence is an important first step for a fully automated colonoscopy analysis tool. Typically, such detection system is built using a large annotated data set of frames with and without polyps, which is expensive to be obtained. In this paper, we introduce a new system that detects frames containing polyps as anomalies from a distribution of frames from exams that do not contain any polyps. The system is trained using a one-class training set consisting of colonoscopy frames without polyps -- such training set is considerably less expensive to obtain, compared to the 2-class data set mentioned above. During inference, the system is only able to reconstruct frames without polyps, and when it tries to reconstruct a frame with polyp, it automatically removes (i.e., photoshop) it from the frame -- the difference between the input and reconstructed frames is used to detect frames with polyps. We name our proposed model as anomaly detection generative adversarial network (ADGAN), comprising a dual GAN with two generators and two discriminators. We show that our proposed approach achieves the state-of-the-art result on this data set, compared with recently proposed anomaly detection systems.
BibTeX
@inproceedings{liu2020photoshopping,
  author = {Yuyuan Liu and Yu Tian and Gabriel Maicas and Leonardo Z. C. T. Pu and Rajvinder Singh and Johan W. Verjans and Gustavo Carneiro},
  title = {{Photoshopping Colonoscopy Video Frames}},
  booktitle = {ISBI},
  year = {2020},
  doi = {10.1109/isbi45749.2020.9098406},
}
Figure