TL;DR
This paper surveys deep weakly-supervised object localization (WSOL) methods for histology image analysis, a critical task hampered by the scarcity of pixel-level annotations. The authors propose a taxonomy of WSOL methods and evaluate representative techniques on colon and breast cancer datasets, revealing significant challenges in localization accuracy and highlighting the need for histology-specific adaptations.
What Problem Does This Paper Address?
Diagnosing cancer from histology images using deep learning (DL) is challenging due to the immense size of whole slide images (WSIs), variations in staining, and the ambiguity of labels. Crucially, accurate cancer grading and localization of regions of interest (ROIs) typically require costly pixel-level annotations, which are difficult and time-consuming to obtain, even for expert pathologists. Weakly-supervised object localization (WSOL) methods offer a promising solution by enabling DL models to learn localization from only image-class labels, thereby reducing the annotation burden. This survey aims to provide a comprehensive overview and comparative analysis of these WSOL methods specifically for histology image analysis.
Methods
The paper introduces a taxonomy to categorize deep WSOL methods into two main families: bottom-up and top-down. Bottom-up methods leverage forward pass information within the network to generate localization maps (Class Activation Maps - CAMs), while top-down methods utilize both forward and backward passes. The survey details various sub-categories within these families, including:
- Spatial pooling: Early methods that used different pooling strategies to derive class scores.
- CAM refinement: Techniques to address CAM under-activation, such as data augmentation, perturbation, self-attention, and pseudo-annotation.
- Features enhancement: Methods that improve localization by learning better features or using shallow features.
- Top-down approaches: Including biologically-inspired methods, gradient-based aggregation, and confidence-based aggregation.
The authors also discuss the critical role of model selection in WSOL, differentiating between selecting models based on classification performance (B-CL) versus localization performance (B-LOC).
Datasets
The authors evaluate the WSOL methods on two public histology datasets:
- GlaS (Colon Cancer): This dataset contains 165 H&E stained histology sections with both pixel-level and image-level annotations for cancer grading.
- CAMELYON16 (Breast Cancer): This dataset comprises 399 WSIs annotated for the presence of metastases. To make these large images usable for WSOL, the authors designed a protocol to sample small sub-images (512x512 pixels) with pixel-wise and image-level annotations.
Key Insights and Ideas
- Histology images are uniquely challenging: Unlike natural images, ROIs in histology data are often less salient, making them harder to detect and leading to higher false negative rates.
- Bottom-up methods dominate: The survey highlights that bottom-up WSOL approaches are currently more active and successful than top-down methods, driving most of the progress in the field.
- Under-activation is a persistent problem: Many WSOL methods struggle to capture the entire object of interest, leading to high false negatives. This is exacerbated by the visual similarity between foreground and background in histology images.
- Over-activation is a new challenge: For histology data, a new behavior of over-activation (leading to false positives) has been observed, likely due to the visual similarity between regions.
- Model selection is critical and antagonistic: Classification and localization tasks are often antagonistic. Selecting a model based on classification performance can lead to poor localization, and vice-versa. This necessitates careful consideration of model selection strategies.
- Histology-specific methods outperform generic ones: Methods specifically designed to address the challenges of histology data (e.g., using priors on ROI size or explicit pixel-wise guidance) generally perform better than generic WSOL methods developed for natural images.
Key Results
- Overall poor localization: The study found that most WSOL methods exhibit poor localization performance on histology datasets, with average PxAP scores significantly lower than fully supervised methods. For instance, on CAMELYON16, the average WSOL PxAP was around 39.01%, compared to over 80% for fully supervised U-Net.
- Dataset difficulty: CAMELYON16 proved to be more challenging than GlaS, with lower average localization performance, attributed to the less distinct nature of metastatic regions.
- Top-down vs. Bottom-up: While top-down methods showed better performance on GlaS (average PxAP 69.08% vs. 60.65% for bottom-up), bottom-up methods were superior on CAMELYON16 (average PxAP 41.09% vs. 32.80% for top-down).
- High false negatives: Confusion matrix analysis revealed a high false negative rate across most WSOL methods, indicating under-activation.
- Model selection impact: Selecting models based on localization performance (B-LOC) yielded slightly better localization results (around 2-8% improvement) compared to classification-based selection (B-CL), though B-CL consistently led to better classification performance.
Limitations and Future Work
- Under-/Over-activation: The paper identifies under-activation (high false negatives) and over-activation (high false positives) as key issues that need to be addressed. Strategies like using priors on region size or pseudo-annotations are discussed as potential solutions.
- Sensitivity to Thresholding: CAMs are shown to be highly sensitive to thresholding, making optimal threshold selection difficult and impacting localization reliability. This remains an open issue for future WSOL method design.
- Model Selection Antagonism: The inherent conflict between classification and localization tasks poses a challenge for model selection. The authors suggest separating these tasks, as explored in recent works, to achieve optimal performance in both.
- Histology-Specific Adaptations: The authors emphasize the need for WSOL methods tailored to the unique characteristics of histology images, such as their size, stain variations, label ambiguity, and the low saliency of ROIs.
- Closing the Performance Gap: A primary future direction is to close the performance gap between weakly and fully supervised methods for histology image analysis. This could involve developing more robust pseudo-labeling strategies, better integration of domain knowledge, and novel architectural designs.
Summary updated