MS-CLAM: Mixed supervision for the classification and localization of tumors in Whole Slide Images

Paul Tourniaire, Marius Ilie, Paul Hofman, Nicholas Ayache, Hervé DelingetteMedical Image Analysis 2023 · DOI

TL;DR

This paper introduces MS-CLAM, a novel deep learning approach that combines slide-level and patch-level labels to improve both tumor classification and localization in whole slide images (WSIs). By leveraging a limited amount of patch-level annotations, MS-CLAM significantly enhances performance and interpretability compared to existing weakly-supervised methods, making it a more efficient and accurate tool for digital pathology.

What Problem Does This Paper Address?

Digital pathology, particularly the analysis of Whole Slide Images (WSIs), presents a significant challenge for deep learning models due to their massive size. Pathologists face a time-consuming task in meticulously annotating these images. While weakly-supervised methods, relying solely on slide-level labels, have emerged, they often struggle with accurate object localization (e.g., identifying tumorous patches). This limitation hinders the interpretability and clinical utility of these models. The paper addresses this gap by proposing a method that effectively utilizes a small amount of patch-level annotations to boost both classification and localization performance, bridging the divide between purely weak supervision and costly full supervision.

Methods

The core of MS-CLAM builds upon the CLAM (Clustering-constrained Attention Multiple Instance Learning) architecture. The key innovations introduced are:

  • Mixed Supervision: MS-CLAM integrates both slide-level (weak) and patch-level (strong) labels. When patch-level labels are available, they are used directly to train the tile classifier. When unavailable, CLAM's pseudo-labeling strategy is employed, but with modifications to address its limitations, particularly for normal slides.
  • Paired Batch Method: To create balanced training batches and improve stability, MS-CLAM proposes processing one tumorous and one normal slide simultaneously. This ensures that the model is exposed to both positive and negative examples within a batch, even when dealing with slides that only contain one type of tissue.
  • Attention Loss: A novel attention loss term is introduced to regularize the attention mechanism. For normal slides, it encourages an even distribution of attention across all tiles, mimicking mean pooling. For tumorous slides, it prioritizes attention on tumorous tiles while still encouraging an even distribution among them, improving both classification and interpretability.
  • Exponential Weighted Sampling (EWSamp): This strategy prioritizes sampling slides with available tile-level labels during training, gradually shifting to uniform sampling as training progresses. This allows the model to effectively learn from the limited strong supervision before leveraging the full dataset.

Datasets

The proposed MS-CLAM model was evaluated on two publicly available datasets:

  • Camelyon16: A dataset for classifying lymph node slides and detecting metastatic regions. It consists of 399 WSIs.
  • DigestPath2019: A dataset for classifying and segmenting tissue in colonoscopy images, comprising 660 images.

Key Insights and Ideas

  • The Power of Limited Strong Supervision: The study demonstrates that even a small percentage of patch-level annotations (as low as 12%) can significantly improve both classification and localization performance, making it a more practical approach than relying solely on expensive full annotations.
  • Bridging the Localization Gap: Existing weakly-supervised methods often sacrifice localization accuracy for classification performance. MS-CLAM effectively addresses this by integrating patch-level supervision, leading to more precise tumor identification.
  • Attention as a Regularizer: The novel attention loss not only guides the model to focus on relevant regions but also improves the interpretability of the attention maps, making it easier to understand why a slide is classified in a certain way.
  • Efficient Training: The proposed methods, particularly the paired batch method and EWSamp, enable efficient training in a single phase, avoiding the need for complex multi-stage training procedures.

Key Results

  • Improved Classification: MS-CLAM consistently outperformed the CLAM baseline in both accuracy and F1-score across both datasets, especially when using even a small percentage of annotated slides. For example, on DigestPath2019, MS-CLAM with 12% annotation improved accuracy by 3.5% and F1-score by 4.2% compared to CLAM MB.
  • Enhanced Localization: MS-CLAM achieved significantly higher Dice scores for tumor localization and much better specificity (fewer false positives) on normal slides compared to baselines. On Camelyon16, MS-CLAM with 0% annotation already achieved a Dice score of 0.331, far exceeding CLAM SB's 0.212.
  • Competitive Performance: With only 62% of annotated slides, MS-CLAM achieved performance comparable to or exceeding that of models trained with 100% annotations, highlighting its efficiency.
  • Interpretability: The attention loss led to more uniform and interpretable attention maps, providing better insights into the model's decision-making process.

Limitations and Future Work

  • Exhaustive Annotations: The current approach assumes that available patch-level annotations are exhaustive. Incomplete annotations could lead to erroneous pseudo-labels and negatively impact performance.
  • Tile-Level Inaccuracy: The square tile format can lead to approximate localization, especially when tumor regions are small or irregular.
  • Binary Classification Focus: The current MS-CLAM model is designed for binary classification.
  • Future Work: The authors suggest extending MS-CLAM to multi-class classification tasks, such as Gleason grading, where slide-level and tile-level labels might differ. They also propose exploring finer-grained tile-level supervision, accounting for the ratio of tumor within a tile rather than just its presence.

Summary updated

BibTeX
@article{tourniaire2023ms,
  author = {Paul Tourniaire and Marius Ilie and Paul Hofman and Nicholas Ayache and Hervé Delingette},
  title = {{MS-CLAM: Mixed supervision for the classification and localization of tumors in Whole Slide Images}},
  journal = {Medical Image Analysis},
  year = {2023},
  doi = {10.1016/j.media.2023.102763},
}
Figure