Transferability of Deep Learning Algorithms for Malignancy Detection in Confocal Laser Endomicroscopy Images from Different Anatomical Locations of the Upper Gastrointestinal Tract

Marc Aubreville, Miguel Goncalves, Christian Knipfer, Nicolai Oetter, Helmut Neumann, Florian Stelzle, Christopher Bohr, Andreas K. MaierBIOSTEC (Selected Papers) 2018 · DOI · OpenReview

TL;DR

This paper explores the transferability of deep learning models for detecting Squamous Cell Carcinoma (SCC) in Confocal Laser Endomicroscopy (CLE) images across different anatomical locations (oral cavity and vocal folds). They propose a novel whole-image classification method using Inception V.3 and demonstrate its effectiveness, showing promising generalization capabilities between these sites.

What Problem Does This Paper Address?

Diagnosing Squamous Cell Carcinoma (SCC), a common epithelial cancer, often relies on invasive biopsies. Confocal Laser Endomicroscopy (CLE) offers a non-invasive alternative for in-vivo diagnosis, but interpreting CLE images requires extensive training, limiting its widespread clinical adoption. To address this, researchers are developing automated detection methods. A key challenge is whether models trained on CLE images from one anatomical region (e.g., oral cavity) can effectively diagnose SCC in another (e.g., vocal folds), a concept known as transfer learning. This paper investigates this transferability, aiming to build more robust and broadly applicable diagnostic tools for head and neck SCC.

Methods

The authors compare two deep learning approaches for SCC detection in CLE images:

  1. Patch-based method (ppf): This existing method involves extracting small patches from the CLE images, classifying each patch using a Convolutional Neural Network (CNN), and then fusing these patch-level predictions to obtain an image-level probability.
  2. Whole Image Classification with Activation Maps (proposed): This novel approach utilizes a pre-trained Inception V.3 network. The images undergo specific preprocessing, including a circular extrapolation to handle the round field of view inherent in CLE. The network is trained on image-level labels. Crucially, during inference, a second branch generates class activation maps, providing insights into which image regions contribute most to the classification. This method aims to leverage the entire image context for diagnosis.

Datasets

The study utilized two primary datasets:

  • Oral Cavity (OC) dataset: Consists of 116 image sequences from 12 subjects with histopathologically confirmed head and neck SCC in the oral cavity. Images were acquired from three non-affected sites (alveolar ridge, lower inner labium, hard palate) and lesion sites.
  • Vocal Cords (VC) dataset: Comprises 47 image sequences from five patients with SCC of the vocal folds. This dataset also includes images from the contralateral, presumably healthy, vocal cord.

The paper also mentions a previous dataset of 73 CLE image sequences for vocal folds, but an erratum clarifies the correct count is 47.

Key Insights and Ideas

  • Whole-image context matters: The proposed Inception V.3 based whole-image classification method generally outperforms the patch-based approach, especially in cases with variations in tissue presentation (e.g., patient 6 in the OC dataset). This suggests that leveraging the global image context is crucial for robust diagnosis.
  • Anatomical differences impact transferability: While models trained on the oral cavity dataset showed promising generalization to the vocal folds, the reverse transfer (VC to OC) was significantly less effective. This is attributed to differences in tissue characteristics, particularly cornification, which are visually and statistically evident in the CLE images.
  • Activation maps offer interpretability: The inclusion of class activation maps in the whole-image approach provides valuable insights into the model's decision-making process, which is important for clinical acceptance and understanding.
  • Data augmentation and transfer learning are key: Utilizing pre-trained networks (like Inception V.3 on ImageNet) and employing techniques like early stopping and data augmentation are essential for mitigating overfitting and improving generalization, especially with limited medical datasets.

Key Results

  • Within-site performance: Both methods achieved comparable results on their respective datasets. The image-based classification showed a slight advantage on the oral cavity dataset (AUC 0.9687 vs. 0.9550 for ppf).
  • Cross-site generalization (OC to VC): The generalization from oral cavity to vocal folds yielded promising results. The image-based classification achieved an accuracy of 91.63% on the OC dataset and 92.63% on a joint OC+VC dataset.
  • Cross-site generalization (VC to OC): Generalization from vocal folds to oral cavity performed significantly worse, particularly for the whole-image classification method.
  • Overall performance: Concatenating both datasets (OC+VC) led to improved overall accuracy for both methods, with the image-based system reaching 92.63%.

Limitations and Future Work

  • Limited dataset size: The authors acknowledge that the limited size of their current datasets restricts definitive conclusions about algorithmic generalization. They are preparing to record a larger and more diverse dataset.
  • Lack of histopathological proof for normal tissue: A major limitation is the absence of histopathological confirmation for the "clinically normal" CLE images, as obtaining such samples from healthy tissue is ethically challenging. They are confident that future data collection will include verified healthy tissue.
  • CLE penetration depth: CLE has limited penetration depth, meaning tumors spreading into the submucosa might be missed. The authors suggest combining CLE with other methods like Raman spectroscopy or Optical Coherence Tomography for improved results.
  • Future work: The primary future direction is to acquire a larger and more diverse dataset to further investigate generalization capabilities and improve robustness. They also aim to explore segmentation tasks to identify cancer regions more precisely.

Summary updated

Abstract

Squamous Cell Carcinoma (SCC) is the most common cancer type of the epithelium and is often detected at a late stage. Besides invasive diagnosis of SCC by means of biopsy and histo-pathologic assessment, Confocal Laser Endomicroscopy (CLE) has emerged as noninvasive method that was successfully used to diagnose SCC in vivo. For interpretation of CLE images, however, extensive training is required, which limits its applicability and use in clinical practice of the method. To aid diagnosis of SCC in a broader scope, automatic detection methods have been proposed. This work compares two methods with regard to their applicability in a transfer learning sense, i.e. training on one tissue type (from one clinical team) and applying the learnt classification system to another entity (different anatomy, different clinical team). Besides a previously proposed, patch-based method based on convolutional neural networks, a novel classification method on image level (based on a pre-trained Inception V.3 network with dedicated preprocessing and interpretation of class activation maps) is proposed and evaluated. The newly presented approach improves recognition performance, yielding accuracies of 91.63% on the first data set (oral cavity) and 92.63% on a joint data set. The generalization from oral cavity to the second data set (vocal folds) lead to similar area-under-the-ROC curve values than a direct training on the vocal folds data set, indicating good generalization.
BibTeX
@inproceedings{aubreville2018transferability,
  author = {Marc Aubreville and Miguel Goncalves and Christian Knipfer and Nicolai Oetter and Helmut Neumann and Florian Stelzle and Christopher Bohr and Andreas K. Maier},
  title = {{Transferability of Deep Learning Algorithms for Malignancy Detection in Confocal Laser Endomicroscopy Images from Different Anatomical Locations of the Upper Gastrointestinal Tract}},
  booktitle = {BIOSTEC (Selected Papers)},
  year = {2018},
  doi = {10.1007/978-3-030-29196-9\_4},
}
Figure