Multiple Instance Learning for WSI: A comparative analysis of attention-based approaches

Martim Afonso, Praphulla Bhawsar, Monjoy Saha, Jonas S. Almeida, Arlindo L. OliveiraJournal of Pathology Informatics 2024 · DOI

TL;DR

This paper compares two attention-based Multiple Instance Learning (MIL) approaches, AMIL and AdMIL, for analyzing Whole Slide Images (WSIs) in cancer diagnosis. They demonstrate that these models can effectively detect tumors and, more importantly, can highlight specific regions of interest (ROIs) that correlate with TP53 mutations, offering potential for improved diagnostic tools.

What Problem Does This Paper Address?

Whole Slide Images (WSIs) are massive digital representations of microscope slides, crucial for cancer diagnosis. However, their sheer size and the fact that diagnostic labels are typically assigned at the slide-level (not for individual microscopic regions) pose significant challenges for AI analysis. This "weakly supervised" nature means models need to learn from slide-level labels while identifying relevant cellular or tissue features within the image. Specifically, this paper tackles two key problems: (1) accurately classifying WSIs for cancer presence, and (2) identifying specific morphological features associated with genetic mutations (like TP53) within these WSIs, which is a less explored area.

Methods

The authors compare two established weakly supervised Multiple Instance Learning (MIL) frameworks: Attention MIL (AMIL) and Additive MIL (AdMIL). Both approaches treat each WSI as a "bag" of smaller image patches ("instances") and use attention mechanisms to learn which patches are most important for the final slide-level classification.

  • AMIL: Uses a weighted average of instance embeddings, where weights are learned by a two-layered neural network. The attention mechanism uses a tanh activation function.
  • AdMIL: Modifies the aggregation step by passing each patch embedding multiplied by its attention score through a "patch classifier" to produce a "patch score." These scores are then summed to produce the final bag score. The original AdMIL uses a LeakyReLU activation in its attention layer.
  • "Our Additive MIL": The researchers propose a modified AdMIL architecture that uses the tanh activation function in its attention layer, similar to AMIL, while retaining AdMIL's additive aggregation strategy. This modification was explored to see if it could improve performance or interpretability.

The core idea is to leverage attention mechanisms not just for classification but also to generate heatmaps that visualize which image regions contribute most to the model's decision, potentially acting as a "virtual stain" for pathologists.

Datasets

The study utilizes datasets from The Cancer Genome Atlas (TCGA):

  • TCGA-LUSC (Lung Squamous Cell Carcinoma): Used for tumor detection. A dataset of 694 slides (347 positive, 347 negative) was curated, with patches extracted at 5x magnification. This magnification was chosen as it's commonly used by pathologists for initial tumor assessment and captures tissue-level information.
  • TCGA-BRCA (Invasive Breast Carcinoma): Used for TP53 mutation detection. A dataset of 662 slides (331 positive, 331 negative) was curated. For this task, the models were evaluated at three different magnifications: 5x, 10x, and 20x. This was done to understand how different levels of morphological detail influence the model's ability to detect mutation-related patterns.

Key Insights and Ideas

  • Tissue-level vs. Cell-level Information: For tumor detection, 5x magnification (tissue-level) was sufficient for good performance, suggesting that models can learn to identify tumors similarly to how pathologists do at this level.
  • Morphology and Mutation Correlation: TP53 mutation detection proved more challenging and was significantly more sensitive to higher magnifications (10x and 20x), where finer cellular morphology is resolved. This highlights the importance of detailed cellular features for understanding genetic alterations.
  • Interpretability as a Feature: The attention mechanisms in these MIL models are not just for classification; they generate heatmaps that can highlight regions of interest (ROIs). This "slide painting" capability is a key insight, offering a way to visualize why a model makes a certain prediction, which is highly valuable for pathologists.
  • Modified AdMIL Performance: The researchers' modified AdMIL architecture (using tanh activation) showed competitive performance, particularly on the TP53 mutation task at 20x magnification, suggesting that architectural tweaks can yield improvements.
  • Attention Sparsity: Different attention mechanisms (e.g., LeakyReLU vs. tanh) lead to varying degrees of "sparsity" in attention scores. Sparser attention might be better for scattered tumors, while broader attention might be needed for more diffuse patterns.

Key Results

  • Tumor Detection: Both AMIL and AdMIL achieved high performance, with Area Under the Receiver Operating Characteristic Curve (AUROC) scores around 0.95-0.97. The original AMIL achieved the best AUROC of 0.971 ± 0.015, closely followed by their modified AdMIL at 0.963 ± 0.020.
  • TP53 Mutation Detection: Performance was significantly lower and highly dependent on magnification.
  • At 5x, performance was poor (AUROC around 0.51-0.60), indicating insufficient detail.
  • At 10x, performance improved (AUROC around 0.54-0.71), with AMIL leading at 0.711 ± 0.033.
  • At 20x, AMIL remained strong (AUROC 0.704 ± 0.040), but their modified AdMIL showed a notable improvement, reaching 0.642 ± 0.029, outperforming the original AdMIL at this level.
  • ROI Visualization: Heatmaps were generated for both tasks. For tumor detection, they showed tissue-level patterns. For TP53 mutation detection, higher magnifications revealed more localized patterns, with their modified AdMIL showing a more focused attention at 20x.

Limitations and Future Work

  • Dataset Noise: The researchers acknowledge that using flash-frozen slides (which can have artifacts) and random sampling of tiles for the TP53 mutation task might have introduced noise, potentially impacting model performance.
  • Expert Evaluation of ROIs: A significant limitation is the lack of qualitative evaluation of the generated heatmaps by expert pathologists. The authors suggest this is crucial for validating the clinical utility of the ROIs.
  • Fine-tuning: The models were evaluated in their original configurations without extensive fine-tuning, which could potentially improve performance.
  • Broader Architecture Comparison: The study focused on two specific MIL architectures. Future work could involve comparing these with other MIL approaches, including those with self-attention mechanisms.
  • Ground Truth for ROIs: Establishing ground truth for ROIs would be a laborious but valuable next step for quantitatively assessing the quality of the attention-based localization.

Summary updated

Abstract

Whole slide images (WSI), obtained by high-resolution digital scanning of microscope slides at multiple scales, are the cornerstone of modern Digital Pathology. However, they represent a particular challenge to artificial intelligence (AI)-based/AI-mediated analysis because pathology labeling is typically done at slide-level, instead of tile-level. It is not just that medical diagnostics is recorded at the specimen level, the detection of oncogene mutation is also experimentally obtained, and recorded by initiatives like The Cancer Genome Atlas (TCGA), at the slide level. This configures a dual challenge: (a) accurately predicting the overall cancer phenotype and (b) finding out what cellular morphologies are associated with it at the tile level. To better understand and address these challenges, two existing weakly supervised Multiple Instance Learning (MIL) approaches were explored and compared: Attention MIL (AMIL) and Additive MIL (AdMIL). These architectures were analyzed on tumor detection (a task where these models obtained good results previously) and TP53 mutation detection (a much less explored task). For tumor detection, we built a dataset from Lung Squamous Cell Carcinoma (TCGA-LUSC) slides, with 349 positive and 349 negative slides. The patches were extracted from 5× magnification. For TP53 mutation detection, we explored a dataset built from Invasive Breast Carcinoma (TCGA-BRCA) slides, with 347 positive and 347 negative slides. In this case, we explored three different magnification levels: 5×, 10×, and 20×. Our results show that a modified additive implementation of MIL matched the performance of reference implementation (AUC 0.96), and was only slightly outperformed by AMIL (AUC 0.97) on the tumor detection task. TP53 mutation was most sensitive to features at the higher applications where cellular morphology is resolved. More interestingly from the perspective of the molecular pathologist, we highlight the possible ability of these MIL architectures to identify distinct sensitivities to morphological features (through the detection of regions of interest, ROIs) at different amplification levels. This ability for models to obtain tile-level ROIs is very appealing to pathologists as it provides the possibility for these algorithms to be integrated in a digital staining application for analysis, facilitating the navigation through these high-dimensional images and the diagnostic process.
BibTeX
@article{afonso2024multiple,
  author = {Martim Afonso and Praphulla Bhawsar and Monjoy Saha and Jonas S. Almeida and Arlindo L. Oliveira},
  title = {{Multiple Instance Learning for WSI: A comparative analysis of attention-based approaches}},
  journal = {Journal of Pathology Informatics},
  year = {2024},
  doi = {10.1016/j.jpi.2024.100403},
}
Figure