TL;DR
Researchers have developed a hybrid deep learning framework that combines digital pathology (Whole Slide Images) and radiology (multiparametric MRIs) to classify glioma subtypes. By using a weakly supervised "prototype selection" method to identify the most representative tissue patches and a confidence-based fusion strategy, the model achieved third place in the CPM-RadPath 2020 global challenge.
What Problem Does This Paper Address?
Classifying gliomas—the most common primary brain tumors—is a high-stakes task. Since 2016, the World Health Organization (WHO) has required a combination of histological (visual) and genotypic (molecular) information for accurate diagnosis.
However, building automated tools for this is difficult due to the label constraint issue. While a patient’s case might be labeled "Glioblastoma," a Whole Slide Image (WSI) is massive (often gigapixels in size), and only a small fraction of that image actually contains the diagnostic features. Most of the slide might be healthy tissue, blood, or stroma. Furthermore, precise "pixel-level" annotations of where the tumor is are rarely available. This paper seeks to bridge the gap between global labels and massive, unannotated imaging data by fusing pathology and radiology.
Methods
The authors propose a two-pronged approach that mirrors how clinical teams work, using a primary pathology-based model and a secondary radiology-based model.
- WSI-Based Approach (Weakly Supervised):
- Patch Sampling: Instead of analyzing the whole slide at once, they extract 300 patches from dense cell regions.
- Prototype Selection: To handle the "label constraint," they use a ResNet50 model to extract features from these patches. They then calculate "density" and "diversity" scores for each patch. This allows them to select "prototypes"—the most representative patches for a specific subtype—while filtering out noise like lymphocytes or red blood cells.
- Inference: New patches are compared against these prototypes using cosine similarity, and a majority vote determines the final classification.
- mpMRI-Based Approach:
- Segmentation: A 3D ResUNet segments the brain into three regions: peritumoral edema, enhancing tumor, and necrosis.
- Classification: These segmented volumes, along with patient age, are fed into a 3D ResNet to predict the subtype.
- Label Agreement (Fusion Strategy): The system treats pathology as the "gold standard." It primarily relies on the WSI result. However, if the WSI model’s confidence score falls below a threshold (0.6), the system "consults" the MRI model to make the final call.
Datasets
- CPM-RadPath 2020: The primary dataset containing 221 training cases with paired WSI and mpMRI data. Subtypes include Astrocytoma (A), Glioblastoma (G), and Oligodendroglioma (O).
- BraTS 2020: A larger dataset of 369 cases used specifically to train the 3D MRI segmentation model, ensuring the classification model had high-quality tumor masks to work with.
Key Insights and Ideas
- Pathology vs. Radiology Priority: Unlike previous "feature-level" fusion methods that blend data early on, this paper uses a "decision-level" fusion. This respects the clinical reality that pathology is the primary diagnostic tool, while radiology provides a vital "second opinion" for ambiguous cases.
- The "Aha" Moment in Fusion: The authors found that WSI models often struggle to distinguish between Astrocytoma and Glioblastoma because they share many morphological features. However, Glioblastomas often show distinct "necrosis" on an MRI. By using the MRI as a tie-breaker when WSI confidence was low, the model successfully corrected cases that would have otherwise been misdiagnosed.
- Weak Supervision via Prototypes: By selecting prototypes based on feature similarity rather than manual masks, the researchers bypassed the need for thousands of hours of manual annotation by pathologists.
Key Results
- Competition Performance: The proposed method ranked 3rd place in the testing phase of the CPM-RadPath 2020 challenge.
- Fusion Benefit: On the validation set, the fusion model achieved an F1-score of 0.886 and a Cohen’s Kappa of 0.801.
- Qualitative Success: The paper highlights cases where the WSI model was uncertain (confidence ~0.5) and leaning toward the wrong subtype, but the MRI model—detecting a lack of necrosis—steered the final diagnosis to the correct, less aggressive subtype.
Limitations and Future Work
- Dependency on Segmentation: The MRI classification relies heavily on the accuracy of the initial 3D segmentation. If the segmentation fails, the classification likely will too.
- Hyperparameter Sensitivity: The model is sensitive to empirical settings, such as the 0.6 confidence threshold and the number of patches extracted.
- Expert Intervention: While largely automated, the "prototype selection" phase still benefits from a professional screening out non-representative samples, meaning it isn't yet a fully "hands-off" pipeline.
- Data Scarcity: Like most medical AI research, the relatively small sample size (221 training cases) remains a hurdle for generalizability.
Summary updated