Ensembling EfficientNets for the Classification and Interpretation of Histopathology Images

Athanasios Kallipolitis, Kyriakos Revelos, Ilias MaglogiannisAlgorithms 2021 · DOI

TL;DR

This paper proposes an ensemble of EfficientNets for classifying histopathology images of breast and colon cancer, achieving improved accuracy. Crucially, it integrates a Grad-CAM based explainability method to visualize why the model makes its predictions, making it more trustworthy for clinical applications.

What Problem Does This Paper Address?

Histopathology, the microscopic examination of tissue, is crucial for diagnosing diseases like cancer. Traditionally, this involves manual review of glass slides, which is time-consuming and subjective. Digital pathology, using Whole Slide Images (WSIs), offers a digital alternative, enabling the application of AI. While deep learning models, particularly Convolutional Neural Networks (CNNs), have shown great promise in classifying these images, two key challenges remain:

  1. Performance Improvement: Achieving consistently high accuracy, especially with complex and varied datasets, is an ongoing goal.
  2. Explainability: Deep learning models are often treated as "black boxes." In healthcare, where decisions have high stakes, understanding why a model makes a particular prediction is vital for trust and adoption by medical professionals. This is particularly challenging for ensemble models, which combine multiple networks.

This paper aims to address both by developing an ensemble classification system that not only improves accuracy but also provides interpretable insights into its decision-making process.

Methods

The core of the proposed methodology is an ensemble classifier built upon EfficientNets. EfficientNets are a family of CNNs known for their efficiency and state-of-the-art performance, achieved through a novel compound scaling method that balances network depth, width, and resolution.

Here's a breakdown of the key methods:

  • Ensembling: Instead of relying on a single EfficientNet, the authors combine three different pretrained EfficientNet models (B1, B2, and B3). These models are trained on ImageNet and then used for feature extraction. Their outputs are concatenated and fed into a fully connected layer for final classification. This ensemble approach leverages the diverse strengths of individual models to improve overall robustness and accuracy.
  • Explainability with Grad-CAM: To address the "black box" problem, the paper integrates a Grad-CAM (Gradient-weighted Class Activation Mapping) based explainability scheme. Grad-CAM highlights the regions in an input image that are most influential for a specific class prediction. The authors adapt this technique to work with their ensemble by concatenating feature maps from the base classifiers before applying Grad-CAM. They also combine it with guided backpropagation for finer-grained visualizations. This allows them to generate heatmaps that visually indicate which parts of the histopathology image led to the model's classification.
  • Transfer Learning: The EfficientNet models are pretrained on the large ImageNet dataset, allowing them to leverage general visual features before being fine-tuned for the specific task of histopathology classification.

Datasets

The study utilizes two publicly available datasets:

  • BreakHis: This dataset contains 7909 microscopic breast tumor tissue images from 82 patients. The images are categorized into benign (adenosis, fibroadenoma, phyllodes tumor, tubular adenoma) and malignant (ductal carcinoma, lobular carcinoma, mucinous carcinoma, papillary carcinoma) types. It includes images at four different magnification factors (40×, 100×, 200×, 400×).
  • Bachs (Colorectal Cancer Dataset): This dataset comprises 100,000 non-overlapping image patches from H&E stained histological images of human colorectal cancer (CRC) and normal tissue. Images are 224x224 pixels and include tissue classes like adipose, background, debris, lymphocytes, mucus, smooth muscle, normal colon mucosa, cancer-associated stroma, and colorectal adenocarcinoma epithelium.

Key Insights and Ideas

  • Ensembling for Accuracy Boost: Combining multiple EfficientNet models, even those with similar underlying architectures but different scaling factors, demonstrably improves classification accuracy compared to individual models, especially when dealing with challenging dataset splits.
  • Explainability is Crucial for Trust: The integration of Grad-CAM and guided backpropagation provides visual explanations that can be evaluated by expert pathologists. This is a significant step towards making AI models more transparent and trustworthy in clinical settings.
  • Grad-CAM Adaptation for Ensembles: The paper successfully adapts Grad-CAM to work with ensemble models by modifying how feature maps are handled, showing that explainability techniques can be extended beyond single network architectures.
  • Generalization Capabilities: The proposed methodology shows promising generalization capabilities, performing well on unseen data from different datasets.

Key Results

  • Classification Accuracy: The ensemble of EfficientNets achieved high accuracy on both the breast and colon cancer datasets. For instance, on the colon cancer dataset with a strict 60-40% train-validation split, the ensemble achieved an accuracy of 0.9822, outperforming single baseline architectures. On the BreakHis dataset, ensemble models consistently showed improved accuracy and AUC across different magnification factors and classification tasks (binary and multiclass).
  • Explainability Performance: Qualitative evaluation by expert pathologists indicated significant accordance between the highlighted regions by the Grad-CAM/guided backpropagation method and the areas considered important for diagnosis. The method generally highlighted epithelial cells, which are relevant for cancer diagnosis.
  • Comparison to Baselines: The ensemble approach demonstrated a minor but consistent improvement over individual baseline architectures, particularly when the dataset splits were made more challenging (e.g., 40-60% or 30-70% splits).

Limitations and Future Work

  • Uniform Morphology Challenge: A key limitation identified is the difficulty in localizing important patterns when the morphology of the disease is uniform across the image. In such cases, the algorithm may assign high importance to less relevant areas or edges.
  • Artefact Removal: The authors suggest that implementing an artefact removal methodology could further enhance the generated explanation results.
  • Fuzziness of Guided Grad-CAM: While guided Grad-CAM offers finer granularity, its visualizations are noted to be fuzzier compared to the more expressive Grad-CAM.
  • Combining Techniques: Future work should focus on combining Grad-CAM with other techniques to better distinguish morphology entities and integrate the coarse-grained properties of Grad-CAM with the strong discrimination abilities of morphological patterns.
  • Further Ensemble Diversity: Exploring different ensemble configurations and diversity-promoting techniques could lead to further performance gains.

Summary updated

Abstract

The extended utilization of digitized Whole Slide Images is transforming the workflow of traditional clinical histopathology to the digital era. The ongoing transformation has demonstrated major potentials towards the exploitation of Machine Learning and Deep Learning techniques as assistive tools for specialized medical personnel. While the performance of the implemented algorithms is continually boosted by the mass production of generated Whole Slide Images and the development of state-of the-art deep convolutional architectures, ensemble models provide an additional methodology towards the improvement of the prediction accuracy. Despite the earlier belief related to deep convolutional networks being treated as black boxes, important steps for the interpretation of such predictive models have also been proposed recently. However, this trend is not fully unveiled for the ensemble models. The paper investigates the application of an explanation scheme for ensemble classifiers, while providing satisfactory classification results of histopathology breast and colon cancer images in terms of accuracy. The results can be interpreted by the hidden layers’ activation of the included subnetworks and provide more accurate results than single network implementations.
BibTeX
@inproceedings{kallipolitis2021ensembling,
  author = {Athanasios Kallipolitis and Kyriakos Revelos and Ilias Maglogiannis},
  title = {{Ensembling EfficientNets for the Classification and Interpretation of Histopathology Images}},
  booktitle = {Algorithms},
  year = {2021},
  doi = {10.3390/a14100278},
}
Figure