The Image Biomarker Standardization Initiative: Standardized Quantitative Radiomics for High-Throughput Image-based Phenotyping

Alex Zwanenburg, Martin Vallières, Mahmoud A. Abdalah, Hugo J.w.l. Aerts, Vincent Andrearczyk, Aditya Apte, Saeed Ashrafinia, Spyridon Bakas, Roelof J. Beukinga, Ronald Boellaard, Marta Bogowicz, Luca Boldrini, Irène Buvat, Gary Cook, Christos Davatzikos, Adrien Depeursinge, Marie-Charlotte Desseroit, N. Dinapoli, Cuong V. Dinh, Sebastian Echegaray, Issam El Naqa, Andriy Fedorov, Roberto Gatta, Robert J. Gillies, Vicky Goh, Michael Götz, Matthias Gückenberger, Sung Min Ha, Mathieu Hatt, Fabian Isensee, Philippe Lambin, Stefan Leger, Ralph T. H. Leijenaar, Jacopo Lenkowicz, Fiona Lippert, Are Losnegård, Klaus Maier‐Hein, Olivier Morin, Henning Müller, Sandy Napel, Christophe Nioche, Fanny Orlhac, Sarthak Pati, Elisabeth Pfaehler, Arman Rahmim, Arvind U K Rao, Jonas Scherer, Muhammad Musib Siddique, Nanna M. Sijtsema, Jairo Socarras Fernandez, Emiliano Spezi, Roel J.h.m. Steenbakkers, Stephanie Tanadini‐Lang, Daniela Thorwarth, Esther G.c. Troost, Taman Upadhaya, Vincenzo Valentini, Lisanne V. van Dijk, Joost J. M. van Griethuysen, Floris H. P. van Velden, P. Whybra, Christian Richter, Steffen LöckRadiology 2020 · DOI

TL;DR

This paper introduces the Image Biomarker Standardization Initiative (IBSI), a crucial effort to standardize radiomics features. By developing a comprehensive set of definitions, reference values, and reporting guidelines, IBSI aims to overcome the reproducibility crisis in radiomics, paving the way for more reliable and clinically applicable quantitative imaging biomarkers.

What Problem Does This Paper Address?

The paper tackles a significant challenge in the field of radiomics: the lack of standardization. While radiomics holds immense promise for extracting quantitative biomarkers from medical images to characterize tumors and predict treatment outcomes, the absence of universally agreed-upon definitions, calculation methods, and reference values has led to a "wild frontier" of irreproducible results. This inconsistency hinders the clinical translation of radiomics findings, making it difficult to compare studies or trust the reliability of extracted features. IBSI aims to bridge this gap by providing a standardized framework for radiomic feature extraction and reporting.

Methods

The IBSI initiative employed a multi-phase approach to standardize a set of radiomic features.

  • Phase I: Focused on standardizing 174 basic radiomic features. 25 research teams independently computed these features from a digital phantom, using their own radiomics software implementations. This phase aimed to establish initial reference values and assess the reproducibility of features without complex image processing.
  • Phase II: Expanded to include 1347 derived features, computed from a lung cancer CT image using predefined image processing configurations. 15 teams participated in this phase, further refining reference values and evaluating features under more realistic clinical imaging conditions.
  • Phase III: A prospective validation phase using a public dataset of multimodality images (CT, PET, MRI) from 51 patients with soft-tissue sarcoma. This phase aimed to assess the reproducibility of the standardized features across different imaging modalities and software implementations.

Consensus on reference values was measured by the frequency of the modal value and classified as weak, moderate, strong, or very strong.

Datasets

The IBSI initiative utilized the following datasets:

  • Digital Phantom: A small, custom-developed digital phantom was used in Phase I. This phantom was designed to be simple, consisting of 5x4x4 voxels with specific grey level intensities and excluded voxels. It served as a baseline for feature extraction without complex image processing.
  • Lung Cancer CT Image: A CT image from one patient with lung cancer (PAT1) was used in Phase II. This dataset, available through cancerdata.org (DOI:10.17195/candat.2016.08.1), was used with predefined image processing configurations.
  • Multimodality Image Dataset: A public dataset of CT, 18F-FDG PET, and T1-weighted MRI images from 51 patients with soft-tissue sarcoma was used in Phase III for prospective validation. This dataset is available via the IBSI GitHub repository.

Key Insights and Ideas

  • Reproducibility Crisis in Radiomics: The paper highlights that the lack of standardization is a major bottleneck for the clinical adoption of radiomics.
  • Importance of Standardized Definitions: IBSI's core contribution is providing standardized definitions and reference values for a large set of radiomic features, enabling consistent interpretation across different studies and software.
  • Multi-Phase Validation: The rigorous multi-phase approach, involving a digital phantom, a clinical CT image, and a multimodality dataset, ensures robust standardization and validation of the features.
  • Impact of Image Processing: The study implicitly emphasizes the significant impact of image processing steps (interpolation, discretization, etc.) on feature values and the need for standardized protocols.
  • Nomenclature System: The development of a clear nomenclature system is crucial for unambiguously referring to specific features and their associated parameters.

Key Results

  • High Consensus Achieved: After the standardization process, IBSI achieved strong or very strong consensus on reference values for a vast majority of features (95.1% in Phase I and 90.6% in Phase II).
  • Standardized Feature Set: A set of 169 radiomics features were successfully standardized.
  • Excellent Reproducibility: In the final validation phase, these standardized features demonstrated excellent reproducibility across different imaging modalities (CT, PET, MRI) and software implementations.
  • Reduced Ambiguity: The proposed nomenclature system effectively addresses the ambiguity arising from different software implementations and feature extraction settings.

Limitations and Future Work

  • Feature Exclusions: While a large number of features were standardized, some features (two in Phase I and 19 in Phase II) still had weak consensus and were not standardized.
  • Ongoing Development: The paper mentions that the IBSI is an ongoing initiative, with potential for future expansion of the feature set and refinement of guidelines.
  • Clinical Validation: While the study focuses on standardization, the ultimate clinical utility and prognostic power of these standardized features require further extensive clinical validation.
  • Software Compliance: The paper introduces the concept of IBSI compliance for radiomics software, suggesting that future work will involve verifying and certifying software implementations against the standardized features.

Summary updated

Abstract

Background Radiomic features may quantify characteristics present in medical imaging. However, the lack of standardized definitions and validated reference values have hampered clinical use. Purpose To standardize a set of 174 radiomic features. Materials and Methods Radiomic features were assessed in three phases. In phase I, 487 features were derived from the basic set of 174 features. Twenty-five research teams with unique radiomics software implementations computed feature values directly from a digital phantom, without any additional image processing. In phase II, 15 teams computed values for 1347 derived features using a CT image of a patient with lung cancer and predefined image processing configurations. In both phases, consensus among the teams on the validity of tentative reference values was measured through the frequency of the modal value and classified as follows: less than three matches, weak; three to five matches, moderate; six to nine matches, strong; 10 or more matches, very strong. In the final phase (phase III), a public data set of multimodality images (CT, fluorine 18 fluorodeoxyglucose PET, and T1-weighted MRI) from 51 patients with soft-tissue sarcoma was used to prospectively assess reproducibility of standardized features. Results Consensus on reference values was initially weak for 232 of 302 features (76.8%) at phase I and 703 of 1075 features (65.4%) at phase II. At the final iteration, weak consensus remained for only two of 487 features (0.4%) at phase I and 19 of 1347 features (1.4%) at phase II. Strong or better consensus was achieved for 463 of 487 features (95.1%) at phase I and 1220 of 1347 features (90.6%) at phase II. Overall, 169 of 174 features were standardized in the first two phases. In the final validation phase (phase III), most of the 169 standardized features could be excellently reproduced (166 with CT; 164 with PET; and 164 with MRI). Conclusion A set of 169 radiomics features was standardized, which enabled verification and calibration of different radiomics software. © RSNA, 2020 <i>Online supplemental material is available for this article.</i> See also the editorial by Kuhl and Truhn in this issue.
BibTeX
@inproceedings{zwanenburg2020image,
  author = {Alex Zwanenburg and Martin Vallières and Mahmoud A. Abdalah and Hugo J.W.L. Aerts and Vincent Andrearczyk and Aditya Apte and Saeed Ashrafinia and Spyridon Bakas and Roelof J. Beukinga and Ronald Boellaard and Marta Bogowicz and Luca Boldrini and Irène Buvat and Gary Cook and Christos Davatzikos and Adrien Depeursinge and Marie-Charlotte Desseroit and N. Dinapoli and Cuong V. Dinh and Sebastian Echegaray and Issam El Naqa and Andriy Fedorov and Roberto Gatta and Robert J. Gillies and Vicky Goh and Michael Götz and Matthias Gückenberger and Sung Min Ha and Mathieu Hatt and Fabian Isensee and Philippe Lambin and Stefan Leger and Ralph T. H. Leijenaar and Jacopo Lenkowicz and Fiona Lippert and Are Losnegård and Klaus Maier‐Hein and Olivier Morin and Henning Müller and Sandy Napel and Christophe Nioche and Fanny Orlhac and Sarthak Pati and Elisabeth Pfaehler and Arman Rahmim and Arvind U K Rao and Jonas Scherer and Muhammad Musib Siddique and Nanna M. Sijtsema and Jairo Socarras Fernandez and Emiliano Spezi and Roel J.H.M. Steenbakkers and Stephanie Tanadini‐Lang and Daniela Thorwarth and Esther G.C. Troost and Taman Upadhaya and Vincenzo Valentini and Lisanne V. van Dijk and Joost J. M. van Griethuysen and Floris H. P. van Velden and P. Whybra and Christian Richter and Steffen Löck},
  title = {{The Image Biomarker Standardization Initiative: Standardized Quantitative Radiomics for High-Throughput Image-based Phenotyping}},
  booktitle = {Radiology},
  year = {2020},
  doi = {10.1148/radiol.2020191145},
}
Figure