Quality control of radiomic features using 3D-printed CT phantoms

Usman Mahmood, Aditya Apte, Christopher Kanan, ‬David D. B. Bates, Giuseppe Corrias, Lorenzo Manneli, Jung Hun Oh, Yusuf E. Erdi, John Nguyen, Joseph O’Deasy, Amita Shukla‐DaveJournal of Medical Imaging 2021 · DOI

TL;DR

This paper introduces a novel method for creating anatomically informed 3D-printed CT phantoms using patient-derived tumor data. These phantoms serve as a crucial tool for validating the robustness and repeatability of radiomic features across different CT scanning protocols, addressing a significant challenge in clinical radiomics.

What Problem Does This Paper Address?

Radiomics, the extraction of quantitative features from medical images, holds immense promise for cancer diagnosis, prognosis, and treatment response prediction. However, a major hurdle to its widespread clinical adoption is the lack of standardization. Radiomic features can vary significantly depending on the CT scanner, acquisition parameters, reconstruction algorithms, and post-processing software used. This variability makes it difficult to compare results across studies and institutions, and raises concerns about the reliability of radiomic biomarkers. Existing quality control (QC) phantoms are often too simplistic, lacking the complex textures and anatomical variations found in real patient tumors, thus failing to adequately assess the impact of these variations on radiomic features.

Methods

The authors developed a process to create custom 3D-printed radiomic phantoms. This involved:

  1. Data Acquisition: Using DICOM CT scans from patients with pancreatic adenocarcinoma, non-small cell lung carcinoma, and hepatic cirrhosis.
  2. Segmentation and Merging: Segmenting tumor regions and liver background from patient scans. These segmented regions were then merged to create a composite anatomical structure.
  3. 3D Printing: Utilizing a multimaterial, ultra-high-resolution 3D printer (PolyJet Objet 260 Connex 3) with voxel printing capabilities. This allowed for the deposition of different resin materials with varying Hounsfield Unit (HU) values, mimicking tissue densities. The process involved converting segmented data into binary raster files that dictated the spatial placement of each material.
  4. Phantom Design: The final phantom was designed as a circular disc (18 cm diameter, 4 cm width) to fit within existing CT QC enclosures.
  5. Scanning and Evaluation: The fabricated phantom was scanned 30 times using a standard clinical CT protocol to assess repeatability. Subsequently, it was scanned under various modified protocols (different kernels, tube potentials, tube currents, ASiR levels, dual-energy CT, voxel sizes, and off-center positioning) to evaluate the reproducibility and sensitivity of radiomic features to these changes.
  6. Radiomic Feature Extraction: Prognostic radiomic features for NSCLC and PDAC were extracted using the CERR software.
  7. Statistical Analysis: Structural Similarity Index (SSIM) was used to compare the phantom to the original liver background. Within-subject coefficient of variation (wCV) was used to assess repeatability, with <1.0% considered repeatable. Percent deviation (pDV) was calculated to quantify the change in radiomic features under different scan modes relative to a baseline protocol.

Datasets

The study utilized anonymized CT scans from six unique patients:

  • Four patients with pancreatic adenocarcinoma (PDAC).
  • One patient with non-small cell lung carcinoma (NSCLC).
  • One patient with advanced hepatic cirrhosis.

These patient scans were used to extract anatomical and textural information for the creation of the 3D-printed phantoms. The study did not introduce a new benchmark dataset but rather used patient data to create a novel physical phantom for evaluation.

Key Insights and Ideas

  • 3D Printing as a Realistic Phantom Tool: The core idea is that 3D printing, especially with multimaterial capabilities, can create phantoms that more accurately mimic the complex textures and anatomical variations of real tumors than traditional homogeneous phantoms. This is crucial for realistic radiomic feature validation.
  • Prognostic Features are Sensitive: Many previously identified "prognostic" radiomic features, while repeatable under a standard protocol, showed significant variability (high pDV) when CT parameters were altered. This highlights that their clinical utility might be limited without careful standardization or exclusion.
  • First-Order Energy is Robust: Interestingly, first-order energy features demonstrated remarkable stability across various scan modes and reconstruction algorithms, suggesting they are less susceptible to technical variations and potentially more reliable for clinical use.
  • Texture Features are More Vulnerable: Gray level co-occurrence matrix (GLCM) features like contrast, dissimilarity, and entropy were particularly sensitive to changes in image noise, voxel size, and reconstruction kernels, often showing deviations greater than 15%.
  • Voxel-Based Printing is Key: The use of voxel printing technology, rather than surface-based methods (like STL), was essential for capturing the internal structure and texture of the segmented regions, leading to more realistic phantoms.

Key Results

  • High Structural Similarity: The SSIM between the original cirrhotic liver background and the 3D-printed phantom was 0.71, indicating a good degree of structural similarity.
  • Excellent Repeatability: The within-subject coefficient of variation (wCV) for all assessed radiomic features was consistently below 1.0% across 30 repeat scans under the reference protocol, demonstrating high repeatability of the phantom itself.
  • Significant Feature Variability: While repeatable, many radiomic features showed substantial percent deviations (pDV) when subjected to different scan modes. For example, GLCM contrast and dissimilarity features in PDAC tumors exceeded 40% and 20% deviation under reduced dose scan modes and dual-energy CT.
  • Specific Feature Sensitivity: Gray level co-occurrence features (contrast, dissimilarity, entropy) were particularly affected by scan mode changes, often showing pDV > ±15%.
  • First-Order Energy Stability: The average pDV for first-order energy was only 0.01% across all scan modes, underscoring its robustness.

Limitations and Future Work

  • Limited Material Properties: The authors acknowledge that the available photopolymer resins have limited HU value ranges. Future work could explore doping agents to expand this range and better mimic a wider spectrum of tissue densities.
  • Phantom Size Constraints: The size of the 3D printer bed limits the maximum anatomical area that can be printed. Designing modular phantoms is suggested to overcome this.
  • No Motion Assessment: The phantom was stationary during imaging, so the impact of patient motion on radiomic features was not assessed.
  • Crude Anthropomorphism: While anatomically informed, the phantom lacks finer details like fat planes and the influence of beam hardening from contrast or bone, making it "crudely anthropomorphic."
  • Future Directions: The authors propose that these custom phantoms are a flexible and practical way to validate and compare radiomic metrics longitudinally, over time, and across different systems. They suggest that such phantoms can be part of a broader QC process for radiomics. Further research could focus on incorporating more realistic material properties and exploring the impact of motion.

Summary updated

Abstract

<b>Purpose</b>: The lack of standardization in quantitative radiomic measures of tumors seen on computed tomography (CT) scans is generally recognized as an unresolved issue. To develop reliable clinical applications, radiomics must be robust across different CT scan modes, protocols, software, and systems. We demonstrate how custom-designed phantoms, imprinted with human-derived patterns, can provide a straightforward approach to validating longitudinally stable radiomic signature values in a clinical setting. <b>Approach</b>: Described herein is a prototype process to design an anatomically informed 3D-printed radiomic phantom. We used a multimaterial, ultra-high-resolution 3D printer with voxel printing capabilities. Multiple tissue regions of interest (ROIs), from four pancreas tumors, one lung tumor, and a liver background, were extracted from digital imaging and communication in medicine (DICOM) CT exam files and were merged together to develop a multipurpose, circular radiomic phantom (18 cm diameter and 4 cm width). The phantom was scanned 30 times using standard clinical CT protocols to test repeatability. Features that have been found to be prognostic for various diseases were then investigated for their repeatability and reproducibility across different CT scan modes. <b>Results</b>: The structural similarity index between the segment used from the patients' DICOM image and the phantom CT scan was 0.71. The coefficient variation for all assessed radiomic features was <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:mo><</mml:mo> <mml:mn>1.0</mml:mn> <mml:mo>%</mml:mo></mml:mrow> </mml:math> across 30 repeat scans of the phantom. The percent deviation (pDV) from the baseline value, which was the mean feature value determined from repeat scans, increased with the application of the lung convolution kernel, changes to the voxel size, and increases in the image noise. Gray level co-occurrence features, contrast, dissimilarity, and entropy were particularly affected by different scan modes, presenting with <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:mi>pDV</mml:mi> <mml:mo>></mml:mo> <mml:mo>±</mml:mo> <mml:mn>15</mml:mn> <mml:mo>%</mml:mo></mml:mrow> </mml:math> . <b>Conclusions</b>: Previously discovered prognostic and popular radiomic features are variable in practice and need to be interpreted with caution or excluded from clinical implementation. Voxel-based 3D printing can reproduce tissue morphology seen on CT exams. We believe that this is a flexible, yet practical, way to design custom phantoms to validate and compare radiomic metrics longitudinally, over time, and across systems.
BibTeX
@article{mahmood2021quality,
  author = {Usman Mahmood and Aditya Apte and Christopher Kanan and ‬David D. B. Bates and Giuseppe Corrias and Lorenzo Manneli and Jung Hun Oh and Yusuf E. Erdi and John Nguyen and Joseph O’Deasy and Amita Shukla‐Dave},
  title = {{Quality control of radiomic features using 3D-printed CT phantoms}},
  journal = {Journal of Medical Imaging},
  year = {2021},
  doi = {10.1117/1.jmi.8.3.033505},
}
Figure