Unsupervised Dual Adversarial Learning for Anomaly Detection in Colonoscopy Video Frames

Yuyuan Liu, Yu Tian, Gabriel Maicas, Leonardo Zorrón Cheng Tao Pu, Rajvinder Singh, Johan Verjans, Gustavo Carneiro

Abstract

The automatic detection of frames containing polyps from a colonoscopy video sequence is an important first step for a fully automated colonoscopy analysis tool. Typically, such detection system is built using a large annotated data set of frames with and without polyps, which is expensive to be obtained. In this paper, we introduce a new system that detects frames containing polyps as anomalies from a distribution of frames from exams that do not contain any polyps. The system is trained using a one-class training set consisting of colonoscopy frames without polyps. During inference, the system is only able to reconstruct frames without polyps, and when it tries to reconstruct a frame with polyp, it automatically removes (i.e., photoshop) it from the frame -- the difference between the input and reconstructed frames is used to detect frames with polyps. We name our proposed model as anomaly detection generative adversarial network (ADGAN), comprising a dual GAN with two generators and two discriminators. We show that our proposed approach achieves the state-of-the-art result on this data set, compared with recently proposed anomaly detection systems.

Introduction

Colorectal cancer is a leading cause of cancer deaths, but early detection of polyps during colonoscopy can significantly improve survival rates. However, manual detection by doctors is imperfect, and automated systems typically require massive, hand-labeled datasets of both "normal" and "polyp" frames. These datasets are expensive and time-consuming to create.

Existing anomaly detection models often struggle with the complex global structure of colonoscopy frames. Simple autoencoders may fail to capture this structure or, conversely, become too proficient at reconstructing everything, including the polyps they are intended to flag. The authors propose a more robust, unsupervised system that distinguishes polyps from healthy tissue without requiring labeled polyp data for training.

Methods: The ADGAN Framework

The authors propose ADGAN, a dual-adversarial learning framework. Unlike a standard GAN, ADGAN utilizes two generators and two discriminators to ensure a consistent mapping between the image space and the latent representation.

The ADGAN architecture features a Visual GAN for image generation and a Latent GAN to ensure consistent encoding during training and testing.
  1. Visual GAN ($G_v, D_v$): Based on the Wasserstein GAN (WGAN) for stability, this pair focuses on generating realistic-looking colonoscopy frames from a latent vector.
  2. Latent GAN ($G_l, D_l$): This pair ensures that the mapping between the image and its hidden representation is consistent. $G_l$ acts as an encoder that maps a real image back into the latent space.

The "Photoshopping" Effect

During training, the model only observes healthy colon frames. During testing, when shown a frame with a polyp, the model cannot represent that polyp in its latent space. Consequently, the reconstruction replaces the polyp with predicted healthy tissue. By calculating the reconstruction error (the pixel-wise difference between input and output), the system generates an "anomaly score" to detect the polyp.

Results and Visual Analysis

The researchers introduced a new colonoscopy dataset of 14,317 high-quality frames. ADGAN achieved a state-of-the-art AUC of 0.7296, outperforming Deep Autoencoders (0.6294), Variational Autoencoders (0.6478), and the previous leading GAN-based method, f-AnoGAN (0.6913).

Visual evidence of the "photoshopping" effect: the top row shows original frames with polyps (circled); the bottom row shows ADGAN's reconstructions where the polyps have been removed.

Qualitative results clearly show the model "healing" the image by removing the polyp in the reconstruction, validating the core hypothesis of the dual-GAN approach. By using GAN discriminators rather than simple MSE losses, ADGAN preserves both fine-grained local textures and the overall global shape of the colon wall.

Conclusion

ADGAN demonstrates that framing polyp detection as an unsupervised anomaly detection problem can eliminate the need for expensive labeled datasets. While the current model was trained on clean frames, future work will focus on handling clinical "distractors" like feces, water, or camera blur, and increasing resolution to detect smaller or flatter polyps.

Summary updated

Abstract

The automatic detection of frames containing polyps from a colonoscopy video sequence is an important first step for a fully automated colonoscopy analysis tool. Typically, such detection system is built using a large annotated data set of frames with and without polyps, which is expensive to be obtained. In this paper, we introduce a new system that detects frames containing polyps as anomalies from a distribution of frames from exams that do not contain any polyps. The system is trained using a one-class training set consisting of colonoscopy frames without polyps -- such training set is considerably less expensive to obtain, compared to the 2-class data set mentioned above. During inference, the system is only able to reconstruct frames without polyps, and when it tries to reconstruct a frame with polyp, it automatically removes (i.e., photoshop) it from the frame -- the difference between the input and reconstructed frames is used to detect frames with polyps. We name our proposed model as anomaly detection generative adversarial network (ADGAN), comprising a dual GAN with two generators and two discriminators. We show that our proposed approach achieves the state-of-the-art result on this data set, compared with recently proposed anomaly detection systems.
BibTeX
@inproceedings{liu2020unsupervised,
  author = {Yuyuan Liu and Yu Tian and Gabriel Maicas and Leonardo Zorrón Cheng Tao Pu and Rajvinder Singh and Johan Verjans and Gustavo Carneiro},
  title = {{Unsupervised Dual Adversarial Learning for Anomaly Detection in Colonoscopy Video Frames}},
  booktitle = {ISBI},
  year = {2020},
  doi = {10.1109/isbi45749.2020.9098406},
}
Figure