A weakly supervised deep learning-based method for glioma subtype classification using WSI and mpMRIs

Wei-Wen Hsu, Jing-Ming Guo, Linmin Pei, Ling-An Chiang, Yao‐Feng Li, Jui-Chien Hsiao, Rivka R. Colen, Peizhong LiuScientific 2022 · DOI

TL;DR

Researchers have developed a hybrid deep learning framework that combines digital pathology (Whole Slide Images) and radiology (multiparametric MRIs) to classify glioma subtypes. By using a weakly supervised "prototype selection" method to identify the most representative tissue patches and a confidence-based fusion strategy, the model achieved third place in the CPM-RadPath 2020 global challenge.

What Problem Does This Paper Address?

Classifying gliomas—the most common primary brain tumors—is a high-stakes task. Since 2016, the World Health Organization (WHO) has required a combination of histological (visual) and genotypic (molecular) information for accurate diagnosis.

However, building automated tools for this is difficult due to the label constraint issue. While a patient’s case might be labeled "Glioblastoma," a Whole Slide Image (WSI) is massive (often gigapixels in size), and only a small fraction of that image actually contains the diagnostic features. Most of the slide might be healthy tissue, blood, or stroma. Furthermore, precise "pixel-level" annotations of where the tumor is are rarely available. This paper seeks to bridge the gap between global labels and massive, unannotated imaging data by fusing pathology and radiology.

Methods

The authors propose a two-pronged approach that mirrors how clinical teams work, using a primary pathology-based model and a secondary radiology-based model.

  1. WSI-Based Approach (Weakly Supervised):
    • Patch Sampling: Instead of analyzing the whole slide at once, they extract 300 patches from dense cell regions.
  2. Prototype Selection: To handle the "label constraint," they use a ResNet50 model to extract features from these patches. They then calculate "density" and "diversity" scores for each patch. This allows them to select "prototypes"—the most representative patches for a specific subtype—while filtering out noise like lymphocytes or red blood cells.
  3. Inference: New patches are compared against these prototypes using cosine similarity, and a majority vote determines the final classification.
  4. mpMRI-Based Approach:
    • Segmentation: A 3D ResUNet segments the brain into three regions: peritumoral edema, enhancing tumor, and necrosis.
  5. Classification: These segmented volumes, along with patient age, are fed into a 3D ResNet to predict the subtype.
  6. Label Agreement (Fusion Strategy): The system treats pathology as the "gold standard." It primarily relies on the WSI result. However, if the WSI model’s confidence score falls below a threshold (0.6), the system "consults" the MRI model to make the final call.

Datasets

  • CPM-RadPath 2020: The primary dataset containing 221 training cases with paired WSI and mpMRI data. Subtypes include Astrocytoma (A), Glioblastoma (G), and Oligodendroglioma (O).
  • BraTS 2020: A larger dataset of 369 cases used specifically to train the 3D MRI segmentation model, ensuring the classification model had high-quality tumor masks to work with.

Key Insights and Ideas

  • Pathology vs. Radiology Priority: Unlike previous "feature-level" fusion methods that blend data early on, this paper uses a "decision-level" fusion. This respects the clinical reality that pathology is the primary diagnostic tool, while radiology provides a vital "second opinion" for ambiguous cases.
  • The "Aha" Moment in Fusion: The authors found that WSI models often struggle to distinguish between Astrocytoma and Glioblastoma because they share many morphological features. However, Glioblastomas often show distinct "necrosis" on an MRI. By using the MRI as a tie-breaker when WSI confidence was low, the model successfully corrected cases that would have otherwise been misdiagnosed.
  • Weak Supervision via Prototypes: By selecting prototypes based on feature similarity rather than manual masks, the researchers bypassed the need for thousands of hours of manual annotation by pathologists.

Key Results

  • Competition Performance: The proposed method ranked 3rd place in the testing phase of the CPM-RadPath 2020 challenge.
  • Fusion Benefit: On the validation set, the fusion model achieved an F1-score of 0.886 and a Cohen’s Kappa of 0.801.
  • Qualitative Success: The paper highlights cases where the WSI model was uncertain (confidence ~0.5) and leaning toward the wrong subtype, but the MRI model—detecting a lack of necrosis—steered the final diagnosis to the correct, less aggressive subtype.

Limitations and Future Work

  • Dependency on Segmentation: The MRI classification relies heavily on the accuracy of the initial 3D segmentation. If the segmentation fails, the classification likely will too.
  • Hyperparameter Sensitivity: The model is sensitive to empirical settings, such as the 0.6 confidence threshold and the number of patches extracted.
  • Expert Intervention: While largely automated, the "prototype selection" phase still benefits from a professional screening out non-representative samples, meaning it isn't yet a fully "hands-off" pipeline.
  • Data Scarcity: Like most medical AI research, the relatively small sample size (221 training cases) remains a hurdle for generalizability.

Summary updated

Abstract

Accurate glioma subtype classification is critical for the treatment management of patients with brain tumors. Developing an automatically computer-aided algorithm for glioma subtype classification is challenging due to many factors. One of the difficulties is the label constraint. Specifically, each case is simply labeled the glioma subtype without precise annotations of lesion regions information. In this paper, we propose a novel hybrid fully convolutional neural network (CNN)-based method for glioma subtype classification using both whole slide imaging (WSI) and multiparametric magnetic resonance imagings (mpMRIs). It is comprised of two methods: a WSI-based method and a mpMRIs-based method. For the WSI-based method, we categorize the glioma subtype using a 2D CNN on WSIs. To overcome the label constraint issue, we extract the truly representative patches for the glioma subtype classification in a weakly supervised fashion. For the mpMRIs-based method, we develop a 3D CNN-based method by analyzing the mpMRIs. The mpMRIs-based method consists of brain tumor segmentation and classification. Finally, to enhance the robustness of the predictions, we fuse the WSI-based and mpMRIs-based results guided by a confidence index. The experimental results on the validation dataset in the competition of CPM-RadPath 2020 show the comprehensive judgments from both two modalities can achieve better performance than the ones by solely using WSI or mpMRIs. Furthermore, our result using the proposed method ranks the third place in the CPM-RadPath 2020 in the testing phase. The proposed method demonstrates a competitive performance, which is creditable to the success of weakly supervised approach and the strategy of label agreement from multi-modality data.
BibTeX
@inproceedings{hsu2022weakly,
  author = {Wei-Wen Hsu and Jing-Ming Guo and Linmin Pei and Ling-An Chiang and Yao‐Feng Li and Jui-Chien Hsiao and Rivka R. Colen and Peizhong Liu},
  title = {{A weakly supervised deep learning-based method for glioma subtype classification using WSI and mpMRIs}},
  booktitle = {Scientific},
  year = {2022},
  doi = {10.1038/s41598-022-09985-1},
}
Figure