Improving Colonoscopy Lesion Classification Using Semi-Supervised Deep Learning

Mayank Golhar, Taylor L. Bobrow, Mirmilad Pourmousavi Khoshknab, Simran Jit, Saowanee Ngamruengphong, Nicholas J. Durr

TL;DR

This paper introduces a semi-supervised deep learning approach for colonoscopy lesion classification, leveraging a "jigsaw puzzle" pretext task to improve performance, especially when labeled data is scarce. The method significantly boosts accuracy and robustness, demonstrating its potential for more reliable and accessible colon cancer screening.

What Problem Does This Paper Address?

Colorectal cancer is a major health concern, and colonoscopy is the gold standard for its detection and prevention. However, accurately classifying lesions during colonoscopy relies heavily on the endoscopist's skill, and the process can be time-consuming and prone to errors. Deep learning has shown promise in medical image analysis, but its effectiveness is often hampered by the scarcity of expertly annotated medical data, which is expensive and time-consuming to acquire. This paper addresses the challenge of improving colonoscopy lesion classification by effectively utilizing both limited labeled data and abundant unlabeled data through semi-supervised learning.

Methods

The core of this work is a semi-supervised learning (SSL) framework that combines a supervised lesion classification task with an unsupervised "jigsaw puzzle" pretext task. The model uses a ResNet-18 architecture as a shared feature encoder.

  • Supervised Phase: The model is trained to classify colonoscopy images into neoplastic/precancerous or non-neoplastic classes using labeled data. A weighted cross-entropy loss is employed to handle class imbalance.
  • Unsupervised Phase: This phase utilizes the entire dataset (labeled and unlabeled) to train the model to solve a jigsaw puzzle. Images are divided into a 3x3 grid, patches are shuffled, and the network learns to predict the original tile order. This task forces the network to learn high-level, global features rather than relying on low-level cues. A weighted cross-entropy loss is used for this permutation prediction.
  • Alternating Training: The supervised and unsupervised phases are trained iteratively, allowing the unsupervised task to enrich the feature representations learned by the encoder, thereby improving performance on the supervised task.
  • Domain Adaptation & Out-of-Distribution Detection: The paper also investigates how SSL impacts a model's ability to generalize to new domains (e.g., different imaging modalities like White Light Imaging vs. Narrow Band Imaging) and to identify samples that are outside the training distribution.

Datasets

The study utilizes a colonoscopy video dataset collected at Johns Hopkins Hospital.

  • Labeled Data: Consists of image-lesion class label pairs derived from patient procedure videos, with ground truth labels from histopathology analysis.
  • Unlabeled Data: Additional video segments without ground truth classification were collected for training the semi-supervised model.
  • Dataset Summary:
    • Total Frames: 6,649
    • Labeled Frames: 4,095 (3,369 Neoplastic, 726 Non-Neoplastic)
    • Unlabeled Frames: 2,554
    • Imaging Modalities: White Light Imaging (WLI) and Narrow Band Imaging (NBI) were used, with WLI often serving as the source domain and NBI as the target domain for domain adaptation experiments.

Key Insights and Ideas

  • Jigsaw Puzzle as a Powerful Pretext Task: The jigsaw puzzle task effectively encourages the network to learn discriminative features, particularly shape-based representations, which are crucial for lesion classification.
  • SSL Excels in Low-Data Regimes: The semi-supervised approach shows a significantly smaller performance drop compared to the baseline when the amount of labeled data is reduced, highlighting its efficiency in data-scarce scenarios.
  • Improved Generalizability and Robustness: SSL not only improves classification accuracy but also enhances the model's ability to adapt to different imaging domains and detect out-of-distribution samples, making it more practical for real-world clinical deployment.
  • Unsupervised Loss Enhances OOD Detection: Combining the unsupervised jigsaw loss with a KL-divergence score significantly improves the model's capability to detect out-of-distribution samples, a critical aspect for reliable AI in healthcare.

Key Results

  • Lesion Classification Performance: The semi-supervised learning model consistently outperformed the baseline (fully supervised) model across various metrics (accuracy, F1 score, sensitivity, specificity, precision) for all tested percentages of labeled data.
  • With only 6.25% labeled data, SSL achieved an accuracy of 66.80% and an F1 score of 0.79, compared to the baseline's 57.04% accuracy and 0.69 F1 score.
  • Using 100% labeled data, SSL provided a boost of 2.85% in accuracy and 0.03 in F1 score over the baseline.
  • Domain Adaptation: The SSL model showed superior performance in domain adaptation, achieving 1.92% higher accuracy, 0.02 higher F1 score, and 0.07 higher sensitivity when tested on NBI images after training on WLI data.
  • Out-of-Distribution Detection: The SSL model achieved an AUROC of 0.71, significantly outperforming the baseline's AUROC of 0.53, indicating a much better ability to distinguish between in-distribution (WLI) and out-of-distribution (NBI) images.

Limitations and Future Work

  • Limitations:
    • The paper notes that the baseline model outperformed SSL on the specificity metric, though sensitivity is prioritized for precancerous lesion classification.
  • The jigsaw puzzle task might not be universally optimal for all types of lesion detection tasks.
  • Future Work:
    • Explore other semi-supervised learning tasks like image colorization or patch prediction.
  • Investigate the combination of multiple proxy tasks in a multi-task learning setup.
  • Expand the evaluation to other colonoscopy challenges such as polyp detection and segmentation.
  • Conduct a deeper analysis of out-of-distribution detection, especially for "harder" near-distribution anomalies.
  • Examine how SSL improvements scale with larger labeled and unlabeled datasets.
  • Further assess domain generalization across different imaging modalities, endoscopes, and resolutions.

Summary updated

Abstract

While data-driven approaches excel at many image analysis tasks, the performance of these approaches is often limited by a shortage of annotated data available for training. Recent work in semi-supervised learning has shown that meaningful representations of images can be obtained from training with large quantities of unlabeled data, and that these representations can improve the performance of supervised tasks. Here, we demonstrate that an unsupervised jigsaw learning task, in combination with supervised training, results in up to a 9.8% improvement in correctly classifying lesions in colonoscopy images when compared to a fully-supervised baseline. We additionally benchmark improvements in domain adaptation and out-of-distribution detection, and demonstrate that semi-supervised learning outperforms supervised learning in both cases. In colonoscopy applications, these metrics are important given the skill required for endoscopic assessment of lesions, the wide variety of endoscopy systems in use, and the homogeneity that is typical of labeled datasets.
BibTeX
@article{golhar2021improving,
  author = {Mayank Golhar and Taylor L. Bobrow and Mirmilad Pourmousavi Khoshknab and Simran Jit and Saowanee Ngamruengphong and Nicholas J. Durr},
  title = {{Improving Colonoscopy Lesion Classification Using Semi-Supervised Deep Learning}},
  journal = {IEEE Access},
  year = {2021},
  doi = {10.1109/access.2020.3047544},
}
Figure