Photoshopping Colonoscopy Video Frames

Yuyuan Liu, Yu Tian, Gabriel Maicas, Leonardo Zorron Cheng Tao Pu, Rajvinder Singh, Johan W. Verjans, Gustavo Carneiro

This paper, titled "Unsupervised Dual Adversarial Learning for Anomaly Detection in Colonoscopy Video Frames," presents a clever approach to medical imaging: instead of teaching a computer what a disease looks like, teach it what "healthy" looks like and let it flag anything that doesn't fit.

TL;DR

Researchers developed ADGAN, a dual-adversarial network that detects colon polyps by learning exclusively from healthy tissue samples. When the model encounters a polyp during testing, it attempts to "photoshop" the image back into a healthy state; the resulting difference between the original and the reconstructed frame serves as a highly accurate anomaly signal.

What Problem Does This Paper Address?

Colorectal cancer is the third leading cause of cancer deaths, but early detection of polyps during colonoscopies can push survival rates above 90%. The problem is that manual detection is difficult and prone to human error.

While deep learning has helped, most existing systems are "2-class classifiers" that require massive, hand-labeled datasets of both healthy tissue and various types of polyps. These datasets are expensive and time-consuming to create. Furthermore, polyps vary wildly in appearance, making it hard to train a model that recognizes every possible variation. The authors tackle this by reframing the task as unsupervised anomaly detection: if you know exactly what a healthy colon looks like, any polyp—no matter how rare its shape—will stand out as an anomaly.

Methods

The authors introduce the Anomaly Detection Generative Adversarial Network (ADGAN). Unlike a standard GAN, which usually just generates images from random noise, ADGAN uses a dual-structure approach to ensure a tight mapping between images and their "latent" (mathematical) representations.

The Architecture

The system consists of four main components:

  1. Visual Generator ($G_v$): Creates a colonoscopy image from a latent vector.
  2. Visual Discriminator ($D_v$): Tries to tell the difference between real healthy images and those created by the generator.
  3. Latent Generator ($G_l$): Takes an image and maps it back into a latent vector.
  4. Latent Discriminator ($D_l$): Ensures the latent vectors produced by the system match the expected distribution.

The "Photoshop" Mechanism

The model is trained only on healthy frames. During inference, it takes a test frame (which might contain a polyp) and tries to reconstruct it. Because the model has never seen a polyp, it lacks the "vocabulary" to recreate one. Consequently, it reconstructs the area where the polyp was as healthy tissue. By calculating the reconstruction error (the pixel-wise difference between the input and the output), the system can pinpoint the exact location of the anomaly.

Datasets

The study utilized a new colonoscopy dataset derived from 18 videos across 15 patients:

  • Training Set: 13,350 "normal" (healthy) images.
  • Validation Set: 100 normal images for hyperparameter tuning.
  • Testing Set: 967 images, consisting of 290 "abnormal" frames (containing polyps) and 677 "normal" frames.
  • Preprocessing: Images were downsampled to 64x64 resolution, and frames containing heavy blur, feces, or water were removed to ensure the model learned from "clean" tissue.

Key Insights and Ideas

  • One-Class Advantage: By training only on healthy data, the model avoids the "needle in a haystack" problem of finding rare polyp types in a training set.
  • Dual Constraints: Standard autoencoders often accidentally learn to reconstruct anomalies too well, which lowers their detection accuracy. By using a dual GAN setup (constraining both the image space and the latent space), ADGAN forces the model to be "biased" toward healthy reconstructions.
  • End-to-End Training: Unlike previous methods (like f-AnoGAN) that require multiple training stages, ADGAN can be trained in a more streamlined, one-stage process.

Key Results

The researchers compared ADGAN against several baselines, including Deep Auto-Encoders (DAE), Variational Auto-Encoders (VAE), and other GAN-based anomaly detectors like OC-GAN and f-AnoGAN.

  • Performance: ADGAN achieved an AUC (Area Under the Curve) of 0.7296, significantly outperforming the next best model, f-AnoGAN (0.6913).
  • Qualitative Success: Visual results (see Fig. 1 in the paper) clearly show the model "erasing" polyps in its reconstructions, leading to high reconstruction errors that make the polyps easy to flag.

Limitations and Future Work

The authors identify a few areas for improvement:

  • Real-world "Messiness": The current model was trained on "clean" frames. In a real clinical setting, colonoscopy videos are often obscured by feces, water, or camera blur. The authors plan to extend the model to handle these "distractors" without flagging them as dangerous anomalies.
  • Resolution: The 64x64 resolution is relatively low. Increasing resolution could help the model detect smaller or more subtle flat polyps.
  • Temporal Data: The current system analyzes frames individually. Future iterations could leverage the temporal consistency of video (how a polyp moves across multiple frames) to further reduce false positives.

Summary updated

Abstract

The automatic detection of frames containing polyps from a colonoscopy video sequence is an important first step for a fully automated colonoscopy analysis tool. Typically, such detection system is built using a large annotated data set of frames with and without polyps, which is expensive to be obtained. In this paper, we introduce a new system that detects frames containing polyps as anomalies from a distribution of frames from exams that do not contain any polyps. The system is trained using a one-class training set consisting of colonoscopy frames without polyps - such training set is considerably less expensive to obtain, compared to the 2-class data set mentioned above. During inference, the system is only able to reconstruct frames without polyps, and when it tries to reconstruct a frame with polyp, it automatically removes (i.e., photoshop) it from the frame - the difference between the input and reconstructed frames is used to detect frames with polyps. We name our proposed model as anomaly detection generative adversarial network (ADGAN), comprising a dual GAN with two generators and two discriminators. To test our framework, we use a new colonoscopy data set with 14317 images, split as a training set with 13350 images without polyps, and a testing set with 290 abnormal images containing polyps and 677 normal images without polyps. We show that our proposed approach achieves the state-of-the-art result on this data set, compared with recently proposed anomaly detection systems.
BibTeX
@inproceedings{liu2020photoshopping,
  author = {Yuyuan Liu and Yu Tian and Gabriel Maicas and Leonardo Zorron Cheng Tao Pu and Rajvinder Singh and Johan W. Verjans and Gustavo Carneiro},
  title = {{Photoshopping Colonoscopy Video Frames}},
  booktitle = {ISBI},
  year = {2020},
  doi = {10.1109/isbi45749.2020.9098406},
}
Figure