Deep weakly-supervised learning methods for classification and localization in histology images: a survey

Jérôme Rony, Soufiane Belharbi, Jose Dolz, Ismail Ben Ayed, Luke McCaffrey, Eric GrangerMachine Learning for Biomedical Imaging 2019 · DOI · OpenReview

TL;DR

This paper surveys deep weakly-supervised object localization (WSOL) methods for histology image analysis, a critical task hampered by the scarcity of pixel-level annotations. The authors propose a taxonomy of WSOL methods and evaluate representative techniques on colon and breast cancer datasets, revealing significant challenges in localization accuracy and highlighting the need for histology-specific adaptations.

What Problem Does This Paper Address?

Diagnosing cancer from histology images using deep learning (DL) is challenging due to the immense size of whole slide images (WSIs), variations in staining, and the ambiguity of labels. Crucially, accurate cancer grading and localization of regions of interest (ROIs) typically require costly pixel-level annotations, which are difficult and time-consuming to obtain, even for expert pathologists. Weakly-supervised object localization (WSOL) methods offer a promising solution by enabling DL models to learn localization from only image-class labels, thereby reducing the annotation burden. This survey aims to provide a comprehensive overview and comparative analysis of these WSOL methods specifically for histology image analysis.

Methods

The paper introduces a taxonomy to categorize deep WSOL methods into two main families: bottom-up and top-down. Bottom-up methods leverage forward pass information within the network to generate localization maps (Class Activation Maps - CAMs), while top-down methods utilize both forward and backward passes. The survey details various sub-categories within these families, including:

  • Spatial pooling: Early methods that used different pooling strategies to derive class scores.
  • CAM refinement: Techniques to address CAM under-activation, such as data augmentation, perturbation, self-attention, and pseudo-annotation.
  • Features enhancement: Methods that improve localization by learning better features or using shallow features.
  • Top-down approaches: Including biologically-inspired methods, gradient-based aggregation, and confidence-based aggregation.

The authors also discuss the critical role of model selection in WSOL, differentiating between selecting models based on classification performance (B-CL) versus localization performance (B-LOC).

Datasets

The authors evaluate the WSOL methods on two public histology datasets:

  • GlaS (Colon Cancer): This dataset contains 165 H&E stained histology sections with both pixel-level and image-level annotations for cancer grading.
  • CAMELYON16 (Breast Cancer): This dataset comprises 399 WSIs annotated for the presence of metastases. To make these large images usable for WSOL, the authors designed a protocol to sample small sub-images (512x512 pixels) with pixel-wise and image-level annotations.

Key Insights and Ideas

  • Histology images are uniquely challenging: Unlike natural images, ROIs in histology data are often less salient, making them harder to detect and leading to higher false negative rates.
  • Bottom-up methods dominate: The survey highlights that bottom-up WSOL approaches are currently more active and successful than top-down methods, driving most of the progress in the field.
  • Under-activation is a persistent problem: Many WSOL methods struggle to capture the entire object of interest, leading to high false negatives. This is exacerbated by the visual similarity between foreground and background in histology images.
  • Over-activation is a new challenge: For histology data, a new behavior of over-activation (leading to false positives) has been observed, likely due to the visual similarity between regions.
  • Model selection is critical and antagonistic: Classification and localization tasks are often antagonistic. Selecting a model based on classification performance can lead to poor localization, and vice-versa. This necessitates careful consideration of model selection strategies.
  • Histology-specific methods outperform generic ones: Methods specifically designed to address the challenges of histology data (e.g., using priors on ROI size or explicit pixel-wise guidance) generally perform better than generic WSOL methods developed for natural images.

Key Results

  • Overall poor localization: The study found that most WSOL methods exhibit poor localization performance on histology datasets, with average PxAP scores significantly lower than fully supervised methods. For instance, on CAMELYON16, the average WSOL PxAP was around 39.01%, compared to over 80% for fully supervised U-Net.
  • Dataset difficulty: CAMELYON16 proved to be more challenging than GlaS, with lower average localization performance, attributed to the less distinct nature of metastatic regions.
  • Top-down vs. Bottom-up: While top-down methods showed better performance on GlaS (average PxAP 69.08% vs. 60.65% for bottom-up), bottom-up methods were superior on CAMELYON16 (average PxAP 41.09% vs. 32.80% for top-down).
  • High false negatives: Confusion matrix analysis revealed a high false negative rate across most WSOL methods, indicating under-activation.
  • Model selection impact: Selecting models based on localization performance (B-LOC) yielded slightly better localization results (around 2-8% improvement) compared to classification-based selection (B-CL), though B-CL consistently led to better classification performance.

Limitations and Future Work

  • Under-/Over-activation: The paper identifies under-activation (high false negatives) and over-activation (high false positives) as key issues that need to be addressed. Strategies like using priors on region size or pseudo-annotations are discussed as potential solutions.
  • Sensitivity to Thresholding: CAMs are shown to be highly sensitive to thresholding, making optimal threshold selection difficult and impacting localization reliability. This remains an open issue for future WSOL method design.
  • Model Selection Antagonism: The inherent conflict between classification and localization tasks poses a challenge for model selection. The authors suggest separating these tasks, as explored in recent works, to achieve optimal performance in both.
  • Histology-Specific Adaptations: The authors emphasize the need for WSOL methods tailored to the unique characteristics of histology images, such as their size, stain variations, label ambiguity, and the low saliency of ROIs.
  • Closing the Performance Gap: A primary future direction is to close the performance gap between weakly and fully supervised methods for histology image analysis. This could involve developing more robust pseudo-labeling strategies, better integration of domain knowledge, and novel architectural designs.

Summary updated

Abstract

Using deep learning models to diagnose cancer from histology data presents several challenges. Cancer grading and localization of regions of interest (ROIs) in these images normally relies on both image- and pixel-level labels, the latter requiring a costly annotation process. Deep weakly-supervised object localization (WSOL) methods provide different strategies for low-cost training of deep learning models. Using only image-class annotations, these methods can be trained to classify an image, and yield class activation maps (CAMs) for ROI localization. This paper provides a review of state-of-art DL methods for WSOL. We propose a taxonomy where these methods are divided into bottom-up and top-down methods according to the information flow in models. Although the latter have seen limited progress, recent bottom-up methods are currently driving much progress with deep WSOL methods. Early works focused on designing different spatial pooling functions. However, these methods reached limited localization accuracy, and unveiled a major limitation -- the under-activation of CAMs which leads to high false negative localization. Subsequent works aimed to alleviate this issue and recover complete object. Representative methods from our taxonomy are evaluated and compared in terms of classification and localization accuracy on two challenging histology datasets. Overall, the results indicate poor localization performance, particularly for generic methods that were initially designed to process natural images. Methods designed to address the challenges of histology data yielded good results. However, all methods suffer from high false positive/negative localization. Four key challenges are identified for the application of deep WSOL methods in histology -- under/over activation of CAMs, sensitivity to thresholding, and model selection.
BibTeX
@inproceedings{rony2019deep,
  author = {Jérôme Rony and Soufiane Belharbi and Jose Dolz and Ismail Ben Ayed and Luke McCaffrey and Eric Granger},
  title = {{Deep weakly-supervised learning methods for classification and localization in histology images: a survey}},
  booktitle = {Machine Learning for Biomedical Imaging},
  year = {2019},
  doi = {10.59275/j.melba.2023-5g54},
}
Figure