Mutual Contrastive Low-rank Learning to Disentangle Whole Slide Image Representations for Glioma Grading

Lipei Zhang, Yiran Wei, Ying Fu, Stephen J. Price, Carola-Bibiane Schönlieb, Chao Li

TL;DR

This paper introduces a novel "Mutual Contrastive Low-rank Learning" (MCL) framework to improve glioma grading by jointly analyzing two types of whole slide images (WSIs): FFPE and frozen sections. By leveraging mutual learning, a normalized contrastive loss (NMC-loss), and a low-rank loss, the method effectively disentangles multi-modal representations, reduces intra-class variance, and enhances inter-class margins, leading to superior grading performance compared to single-modality or mixed-modality approaches.

What Problem Does This Paper Address?

Glioma grading, crucial for treatment planning, traditionally relies on manual histology assessment of whole slide images (WSIs). While automated grading using WSIs promises faster and more accessible diagnostics, current methods often face challenges. Specifically, models trained on formalin-fixed paraffin-embedded (FFPE) sections can be affected by artifacts introduced during tissue processing. Conversely, frozen sections, while preserving natural cellular morphology, can suffer from lower quality. Existing approaches typically focus on a single modality, potentially missing complementary information and introducing biases. This paper aims to overcome these limitations by developing a method that effectively integrates information from both FFPE and frozen sections to achieve more robust and accurate glioma grading.

Methods

The core of this work is the proposed Mutual Contrastive Low-rank Learning (MCL) scheme. It employs a parallel mutual learning strategy where two separate neural networks (Network $\alpha_1$ and $\alpha_2$) process FFPE and frozen sections, respectively, without weight sharing.

Key novel components include:

  • Normalized Multi-modal Contrastive Loss (NMC-loss): This loss function is designed to disentangle complementary representations from different modalities (FFPE and frozen) within the same latent space. It uses layer normalization to handle the differing distributions of the two modalities, promoting better alignment and feature extraction.
  • Low-rank (LR) Loss: This loss encourages the latent vectors from the same class to lie in a linear subspace by minimizing the matrix rank. It also promotes orthogonality between inter-class latent vectors. The goal is to reduce intra-class variance and increase inter-class margins, leading to more discriminative representations.

The overall optimization combines these losses with a standard classification loss (cross-entropy with Taylor Softmax for label smoothing) for each modality.

Datasets

The study utilized the Cancer Genome Atlas (TCGA) dataset, specifically focusing on glioblastoma (GBM) and lower-grade glioma (LGG) cases. The dataset comprised 499 patients with available FFPE and frozen sections. For pre-processing, WSIs were transformed, patched into 500x500 instances, and redundant patches were removed. The final dataset included 1,680,714 patches for FFPE and 483,886 for frozen sections. The dataset was split into 319 patients for training, 80 for validation, and 100 for testing. Data augmentation techniques were applied to increase sample size.

Key Insights and Ideas

  • Synergy of Modalities: The paper strongly suggests that combining FFPE and frozen sections provides complementary information that is crucial for accurate glioma grading, outperforming single-modality approaches.
  • Disentanglement is Key: The success of the MCL scheme hinges on its ability to disentangle representations. The NMC-loss is specifically designed to achieve this across modalities, while the LR loss further refines this by organizing the latent space.
  • Addressing Intra-class Variance: Glioma, being a heterogeneous tumor type, presents significant intra-class variance. The LR loss is a novel way to tackle this by enforcing a structured latent space, which is often a challenge for standard contrastive methods.
  • Layer Normalization for Heterogeneous Data: Using layer normalization instead of batch normalization in the contrastive loss is critical for handling the distinct distributions of FFPE and frozen sections, a common issue in multi-modal learning.

Key Results

The proposed MCL scheme achieved superior performance across various metrics (Accuracy, Precision, Recall) for both FFPE and frozen sections compared to single training and mixed training approaches.

  • Overall Performance: MCL achieved an accuracy of 0.76 for FFPE and 0.74 for frozen sections, outperforming single training (0.71/0.70) and mixed training (0.70/0.68).
  • Comparison with SOTAs: When combined with state-of-the-art methods like A-MIL, TransMIL, and CLAM, MCL consistently improved their performance, demonstrating its effectiveness in enhancing existing MIL frameworks. For instance, MCL+CLAM achieved 0.79 accuracy on FFPE and 0.75 on frozen sections.
  • Loss Function Effectiveness: The paper demonstrated that the combination of NMC-loss and LR loss yielded the best results, outperforming other contrastive loss functions like KL-loss, marginal triplet loss, and NT-Xent loss.
  • Visualization: UMAP visualizations showed that the MCL scheme effectively disentangled latent vectors, leading to more clustered representations for different tumor grades, indicating better separation and interpretability. Heatmaps also showed the model focusing on relevant proliferation regions.

Limitations and Future Work

  • Temperature Hyper-parameter: The study noted that the training can collapse if the temperature hyper-parameter in the NMC-loss is set too low (below 0.05). While the models showed stable accuracy across different temperatures (0.1 to 1), further investigation into optimal temperature selection might be beneficial.
  • Generalizability: While the paper suggests the scheme can be generalized to other MIL systems and cross-modality learning tasks, further validation on different datasets and cancer types would be valuable.
  • Computational Cost: Mutual learning and contrastive methods can be computationally intensive. While not explicitly detailed as a limitation, future work could explore more efficient implementations or approximations.

Summary updated

Abstract

Whole slide images (WSI) provide valuable phenotypic information for histological assessment and malignancy grading of tumors. The WSI-based grading promises to provide rapid diagnostic support and facilitate digital health. Currently, the most commonly used WSIs are derived from formalin-fixed paraffin-embedded (FFPE) and Frozen section. The majority of automatic tumor grading models are developed based on FFPE sections, which could be affected by the artifacts introduced by tissue processing. The frozen section exists problems such as low quality that might influence training within single modality as well. To overcome this problem in a single modal training and achieve better multi-modal and discriminative representation disentanglement in brain tumor, we propose a mutual contrastive low-rank learning (MCL) scheme to integrate FFPE and frozen sections for glioma grading. We first design a mutual learning scheme to jointly optimize the model training based on FFPE and frozen sections. In this proposed scheme, we design a normalized modality contrastive loss (NMC-loss), which could promote to disentangle multi-modality complementary representation of FFPE and frozen sections from the same patient. To reduce intra-class variance, and increase inter-class margin at intra- and inter-patient levels, we conduct a low-rank (LR) loss. Our experiments show that the proposed scheme achieves better performance than the model trained based on each single modality or mixed modalities and even improves the feature extraction in classical attention-based multiple instances learning methods (MIL). The combination of NMC-loss and low-rank loss outperforms other typical contrastive loss functions.
BibTeX
@inproceedings{zhang2022mutual,
  author = {Lipei Zhang and Yiran Wei and Ying Fu and Stephen J. Price and Carola-Bibiane Schönlieb and Chao Li},
  title = {{Mutual Contrastive Low-rank Learning to Disentangle Whole Slide Image Representations for Glioma Grading}},
  booktitle = {BMVC},
  year = {2022},
  url = {https://openreview.net/forum?id=TpyI3mJpgqh},
}
Figure