<scp>Reproducible Radiomics Features from Multi‐MRI‐Scanner Test–Retest‐Study: Influence on Performance and Generalizability of Models</scp>

Markus Wennmann, Lukas T. Rotkopf, Fabian Bauer, Thomas Hielscher, Jessica Kächele, K. Elias, Niels Weinhold, Marc‐Steffen Raab, Hartmut Goldschmidt, Tim Frederik Weber, Heinz‐Peter Schlemmer, Stefan Delorme, Klaus Maier‐Hein, Peter NeherJournal of Magnetic Resonance Imaging 2024 · DOI

TL;DR

Radiomics models often fail to generalize to new data, hindering clinical adoption. This study investigates whether using only "reproducible" radiomics features, identified through multi-scanner test-retest experiments, improves model performance and generalizability. While reproducible features improved some models, they did not universally enhance performance or guarantee better generalizability, highlighting the complexity of achieving robust radiomics in clinical practice.

What Problem Does This Paper Address?

Radiomics, the extraction of quantitative features from medical images, holds great promise for improving disease diagnosis, prognosis, and treatment response prediction. However, a major hurdle to its clinical translation is the poor generalizability of radiomics models. Models trained on data from one institution or scanner often perform poorly when applied to data from different sources. This is largely attributed to the sensitivity of radiomics features to variations in image acquisition parameters (e.g., scanner type, field strength, sequence parameters).

To address this, current recommendations suggest focusing on radiomics features that are robust and reproducible across different imaging settings. This paper aims to evaluate whether using only such "reproducible" features, identified from a prior in-vivo multi-MRI-scanner test-retest study, can improve the performance and generalizability of machine learning models for predicting plasma cell infiltration from MRI scans.

Methods

The study employed a retrospective design, using data from patients with monoclonal plasma cell disorders. They trained radiomics machine learning models using three different feature selection strategies:

  1. All Features: Utilizing all extracted radiomics features.
  2. Reproducible Features: A subset of 31 features identified as reproducible (Intraclass Correlation Coefficient (ICC) ≥ 0.8) across multiple MRI scanners and sequences in a prior study.
  3. Repeatable Features: A subset of 75 features that showed reproducibility in a simpler test-retest experiment on the same scanner.

Additionally, two calculative feature selection methods, LASSO and MRMR, were used to select 31 features for comparison. Three different machine learning models were trained for each feature subset:

  • Linear Regressor (LR)
  • Support Vector Regressor (SVR)
  • Random Forest Regressor (RFR)

Models were trained on data from "center 1" and then evaluated on an internal test set (also from center 1) and an external, multicentric test set comprising data from seven different centers (centers 2-8). Performance was assessed using Pearson correlation coefficient ($r$) and Mean Absolute Error (MAE) between predicted and actual plasma cell infiltration. Generalizability was quantified by the relative change in performance (Δr and ΔMAE) between the internal and external test sets.

Datasets

  • Training Set: 117 MRIs from center 1.
  • Internal Test Set: 42 MRIs from center 1.
  • External Test Set: 143 MRIs from centers 2-8 (comprising data from eight different MRI scanners across three vendors).

The data was reused from a previous study and included whole-body MRI scans of patients with monoclonal plasma cell disorders, with plasma cell infiltration determined by bone marrow biopsy.

Key Insights and Ideas

  • Reproducibility is not a silver bullet: While using reproducible features improved the external performance of some models (SVR), it did not automatically lead to better performance for all models, and in some cases (RFR), it even led to a decline.
  • Model dependency is crucial: The impact of feature selection strategies (including reproducibility) varied significantly depending on the machine learning model used. What worked for SVR didn't necessarily work for RFR.
  • Generalizability vs. Performance Trade-off: Even when reproducible features improved generalizability (i.e., reduced the performance drop from internal to external testing), this did not always translate to a better absolute external performance, especially if the initial internal performance was lower.
  • Non-reproducible features can still be valuable: Interestingly, features selected by LASSO and MRMR (which included many texture features not found to be reproducible in the prior study) showed comparable or even slightly better external performance for the RFR model compared to using only reproducible features. This suggests that features that are not perfectly stable across scanners might still contribute significantly to predictive power.
  • Complexity of clinical translation: The study underscores that achieving robust and generalizable radiomics models for clinical application is a complex challenge, involving not just feature selection but also the choice of ML model and potentially other factors not explored here.

Key Results

  • SVR Model: Using only reproducible features significantly improved the SVR model's performance on the external test set (r = 0.43 vs. 0.18, MAE = 22.6 vs. 28.2) compared to using all features.
  • RFR Model: The RFR model's performance on the external test set deteriorated when using only reproducible features compared to all features (r = 0.33 vs. 0.44, P = 0.29; MAE = 21.9 vs. 20.5, P = 0.10).
  • Generalizability: For all models, using only reproducible features generally led to better generalizability (less performance drop from internal to external testing) compared to using all features.
  • Overall Best Model: Across all combinations, a RFR model using all features achieved the best external performance (r = 0.44, MAE = 20.5).

Limitations and Future Work

  • Retrospective Design: The study was retrospective, which inherently limits causal inference.
  • Limited 3T Data: While the reproducibility experiments included 3T scanners, the external test set had very few MRIs acquired at 3T, potentially limiting the direct applicability of findings related to 3T scanners.
  • Regression Task Focus: The study focused on a regression task (predicting plasma cell infiltration percentage). Future work should investigate if reproducible features improve performance in classification tasks.
  • Definition of Reproducibility: The study used a specific definition of reproducibility (ICC ≥ 0.8). Future research could explore different statistical metrics and cutoff values for defining "stable" features.
  • Other MRI Sequences/Modalities: The study was limited to T1-weighted turbo spin echo sequences. Future work should explore other MRI sequences and multimodal approaches.
  • Tumor Entity Specificity: Radiomic feature stability can vary by tumor type. Further studies across different entities and modalities are needed.

Summary updated

Abstract

Stage 2.
BibTeX
@article{wennmann2024scp,
  author = {Markus Wennmann and Lukas T. Rotkopf and Fabian Bauer and Thomas Hielscher and Jessica Kächele and K. Elias and Niels Weinhold and Marc‐Steffen Raab and Hartmut Goldschmidt and Tim Frederik Weber and Heinz‐Peter Schlemmer and Stefan Delorme and Klaus Maier‐Hein and Peter Neher},
  title = {{<scp>Reproducible Radiomics Features from Multi‐MRI‐Scanner Test–Retest‐Study: Influence on Performance and Generalizability of Models</scp>}},
  journal = {Journal of Magnetic Resonance Imaging},
  year = {2024},
  doi = {10.1002/jmri.29442},
}
Figure