MGMT promoter methylation status prediction using MRI scans? An extensive experimental evaluation of deep learning models

Numan Saeed, Muhammad Ridzuan, Hussain Alasmawi, Ikboljon Sobirov, Mohammad Yaqub

TL;DR

This paper provides a rigorous "reality check" for the field of radiogenomics by demonstrating that state-of-the-art deep learning models cannot reliably predict MGMT promoter methylation—a critical biomarker for glioblastoma treatment—using MRI scans. Despite previous claims of high accuracy in the literature, this extensive evaluation shows that current models perform barely better than a coin flip (AUC ~0.60), suggesting that the "signal" may simply not be present in standard imaging.

What Problem Does This Paper Address?

Glioblastoma multiforme (GBM) is a devastating brain cancer. A patient's response to chemotherapy (specifically temozolomide) depends heavily on the MGMT promoter methylation status. If the promoter is methylated, the tumor is more sensitive to treatment; if not, the drug is often ineffective.

Currently, determining this status requires an invasive and risky tissue biopsy. The emerging field of radiogenomics aims to replace this with "virtual biopsies"—using deep learning (DL) to predict genetic features directly from non-invasive MRI scans. However, the research community is divided: some studies claim near-perfect accuracy, while others report total failure. This paper seeks to settle the debate through an exhaustive, transparent evaluation using one of the largest public datasets available.

Methods

The authors didn't just test one model; they threw the entire modern DL toolkit at the problem:

  • Architectures: They evaluated Convolutional Neural Networks (CNNs) including various flavors of ResNet, DenseNet, and EfficientNet, as well as modern Vision Transformers (ViT and SwinViT).
  • Multi-modal Fusion: They tested models on individual MRI sequences (T1w, T1wCE, T2, FLAIR) and combinations of them.
  • Pre-training: They explored both ImageNet pre-training and Self-Supervised Learning (SimCLR) to see if better feature representations could unlock the secret.
  • Interpretability Tools: To understand why models were failing, they used Grad-CAM (visualizing where the model looks), Occlusion Sensitivity, and t-SNE/PCA feature visualizations.
  • Loss Landscape Analysis: They visualized the mathematical "terrain" the models navigate during training to see if they were actually finding a meaningful solution or just wandering in a plateau.

Datasets

  • RSNA-ASNR-MICCAI BraTS 2021: The primary dataset, consisting of multi-parametric MRIs from 585 patients. This is significantly larger and more standardized than many datasets used in previous studies that claimed high success.
  • NoduleMNIST3D: A lung nodule malignancy dataset used as a "control." The authors trained the same models on this dataset to prove that their architectures could learn medical features when a clear signal actually exists.

Key Insights and Ideas

  • The "Right Place, Wrong Signal" Phenomenon: Grad-CAM and occlusion maps showed that the models were correctly looking at the tumor area. However, the features they extracted from that area were not discriminative. The models identified where the tumor was, but couldn't see what its genetic status was.
  • The 0.69 Loss Trap: The authors found that the Binary Cross-Entropy (BCE) loss consistently plateaued at approximately 0.69. Mathematically, this is the loss value you get when a model simply guesses 50/50 on a balanced dataset. It’s a clear sign of a model that has learned nothing.
  • Architecture Doesn't Matter (Here): Whether using a simple ResNet-10 or a massive Swin Transformer, the results were essentially the same. This suggests the bottleneck isn't the "brain" of the AI, but the "eyes"—the MRI data itself may lack the necessary resolution or contrast to show MGMT status.
  • A Warning on Data Leakage: The authors suggest that previous studies claiming 90%+ accuracy may have suffered from small sample sizes, lack of independent test sets, or subtle data leakage (where information from the test set accidentally "seeps" into the training process).

Key Results

  • Performance Ceiling: Across dozens of experiments, the Area Under the Curve (AUC) consistently saturated around 0.57 to 0.62. For context, an AUC of 0.50 is random chance.
  • Control Success: When the same ResNet-10 model was applied to the lung nodule dataset (NoduleMNIST3D), it achieved an AUC of 0.856, proving the model architecture was functional and the failure on brain tumors was data-specific.
  • Feature Entanglement: t-SNE visualizations showed that methylated and unmethylated samples remained completely "entangled" in the feature space, even in the final layers of the models.

Limitations and Future Work

The primary limitation is the inherent nature of MRI. Standard clinical MRI sequences might not capture the microscopic biological changes associated with MGMT methylation.

Future directions suggested by the authors include: 1. Multi-modal Integration: Combining MRI with clinical metadata (age, gender) or other "omics" data (proteomics, metabolomics). 2. Advanced Sequences: Exploring more specialized imaging like Diffusion Tensor Imaging (DTI) or MR Spectroscopy. 3. Rigorous Validation: The authors call for the mandatory use of external cohort validation—testing a model on data from a completely different hospital—to prevent the publication of over-optimistic but non-generalisable results.

Summary updated

Abstract

The number of studies on deep learning for medical diagnosis is expanding, and these systems are often claimed to outperform clinicians. However, only a few systems have shown medical efficacy. From this perspective, we examine a wide range of deep learning algorithms for the assessment of glioblastoma - a common brain tumor in older adults that is lethal. Surgery, chemotherapy, and radiation are the standard treatments for glioblastoma patients. The methylation status of the MGMT promoter, a specific genetic sequence found in the tumor, affects chemotherapy's effectiveness. MGMT promoter methylation improves chemotherapy response and survival in several cancers. MGMT promoter methylation is determined by a tumor tissue biopsy, which is then genetically tested. This lengthy and invasive procedure increases the risk of infection and other complications. Thus, researchers have used deep learning models to examine the tumor from brain MRI scans to determine the MGMT promoter's methylation state. We employ deep learning models and one of the largest public MRI datasets of 585 participants to predict the methylation status of the MGMT promoter in glioblastoma tumors using MRI scans. We test these models using Grad-CAM, occlusion sensitivity, feature visualizations, and training loss landscapes. Our results show no correlation between these two, indicating that external cohort data should be used to verify these models' performance to assure the accuracy and reliability of deep learning systems in cancer diagnosis.
BibTeX
@article{saeed2023mgmt,
  author = {Numan Saeed and Muhammad Ridzuan and Hussain Alasmawi and Ikboljon Sobirov and Mohammad Yaqub},
  title = {{MGMT promoter methylation status prediction using MRI scans? An extensive experimental evaluation of deep learning models}},
  journal = {Medical Image Analysis},
  year = {2023},
  doi = {10.1016/j.media.2023.102989},
}
Figure