Reproducibility of findings in modern PET neuroimaging: insight from the NRM2018 grand challenge

Mattia Veronese, Gaia Rizzo, Martín A. Belzunce, Julia Schubert, Graham E. Searle, Alex Whittington, Ayla Mansur, Joel Dunn, Andrew J. Reader, Roger N. Gunn, and the Grand Challenge ParticipantsJournal of Cerebral Blood Flow & Metabolism 2021 · DOI

TL;DR

This paper tackles the critical issue of reproducibility in PET neuroimaging by organizing a grand challenge where 14 teams analyzed the same simulated dataset. While diverse methods were used, the results showed remarkable consistency, suggesting that sharing detailed analysis pipelines is as crucial as sharing data for enhancing confidence in PET findings.

What Problem Does This Paper Address?

The field of neuroimaging, like many scientific disciplines, is grappling with a "reproducibility crisis." A major contributor to this is the lack of standardized pipelines for processing complex data and the often incomplete descriptions of analytical methods in published research. This variability in approaches, even when analyzing the same data, can lead to discrepant results and undermine the reliability of scientific findings. While this issue is well-documented in fMRI, this paper specifically investigates its impact on Positron Emission Tomography (PET) neuroimaging, a field with inherently complex data and flexible analytical procedures.

Methods

The core of this research was the "NRM2018 PET Grand Challenge." Fourteen international teams were tasked with analyzing a simulated PET neurotransmitter release experiment dataset. The dataset was designed to mimic real-world PET scans but with a known "ground truth" for receptor binding changes. Participants were given minimal information about the tracer and target, forcing them to rely on their standard analysis pipelines. The challenge involved identifying areas and quantifying the magnitude of receptor binding changes. The analysis of the submitted solutions focused on two main criteria: the accuracy of the estimated binding potential (BPND) values and the spatial identification of simulated displaced regions.

Datasets

The study utilized a simulated PET dataset created to mimic a neurotransmitter release experiment. This dataset was derived from real PET scans of five healthy subjects, with kinetic parameters and ground truth values for tracer displacement known. The simulation process involved generating time-activity curves (TACs), creating 4D digital phantoms, forward projecting them into sinogram space, and then reconstructing them to simulate realistic PET data. This approach allowed for a controlled evaluation of different analysis methods against a known ground truth, differing from real-world studies where ground truth is often unknown.

Key Insights and Ideas

  • Consistency Amidst Diversity: Despite using a wide array of quantification methods, statistical approaches, and software, the participating teams produced remarkably consistent results in identifying areas of tracer binding displacement. This suggests that the fundamental patterns of change are detectable across different analytical strategies.
  • Beyond Data Sharing: The findings strongly advocate that simply sharing PET data is insufficient to ensure reproducibility. The detailed description and sharing of analysis pipelines and procedures are equally, if not more, critical for others to replicate and build upon research.
  • The Importance of Spatial Smoothing: A common element among the best-performing solutions was the use of spatial filters to smooth the data prior to quantification, indicating its role in managing data noise and improving performance.
  • Bias vs. Identification: While quantification bias (differences between estimated and true values) varied significantly, it didn't always translate to poor identification of displacement regions. A consistent bias across scans within a subject was found to be less detrimental for assessing changes than a non-constant bias.
  • Statistical Power Matters: The study highlighted that smaller regions with lower levels of simulated displacement were consistently undetected by most groups, suggesting that the study might have been underpowered for these specific regions.

Key Results

  • High Agreement on Displacement Areas: 13 out of 14 groups successfully identified significant clusters in the largest simulated displacement regions (ROI 1 and 2). Even for a medium-sized region (ROI 3), 8 out of 14 groups were successful.
  • Low False Positive Rate: Participants generally performed well in minimizing false positives, with an average rate of around 17%±23% of true positive voxels.
  • Quantification Error: The percentage root-mean-squared error (RMSE) for baseline BPND estimates across groups averaged 56%±37%, while RMSE for displacement (ABPND) averaged 33%±17%.
  • Jaccard Similarity: The Jaccard similarity index, measuring the overlap between identified and true displacement regions, showed that the best performing solutions reached around 40%, while the worst were around 1%.
  • Association Between Methods and Performance: Analysis of variance indicated that quantification settings, particularly the implementation of smoothing and the choice of quantification method, significantly influenced performance (RMSE BPND).

Limitations and Future Work

The authors acknowledge several limitations:

  • Simulated Data: The use of simulated data, while providing ground truth, may not perfectly reflect the complexities and noise patterns of real-world PET scans.
  • Simplified Scenario: The simulation omitted certain real-world challenges like motion artifacts and arterial blood input functions, which could have led to greater variability if included.
  • Limited Domain Knowledge: Participants were intentionally blinded to specific tracer details, which is not typical for real-world research where such knowledge informs analysis.
  • Underpowered Regions: The study may have been underpowered to reliably detect displacement in smaller regions.
  • Limited Methodological Scope: Only a subset of quantification methods was implemented, and the participation represented only a fraction of the PET community.

Future work could involve challenges with real data, incorporating arterial input functions, and exploring blood-based quantification methods. The authors strongly advocate for the development and use of standardized checklists for reporting PET data analysis methods to improve transparency and reproducibility.

Summary updated

Abstract

The reproducibility of findings is a compelling methodological problem that the neuroimaging community is facing these days. The lack of standardized pipelines for image processing, quantification and statistics plays a major role in the variability and interpretation of results, even when the same data are analysed. This problem is well-known in MRI studies, where the indisputable value of the method has been complicated by a number of studies that produce discrepant results. However, any research domain with complex data and flexible analytical procedures can experience a similar lack of reproducibility. In this paper we investigate this issue for brain PET imaging. During the 2018 NeuroReceptor Mapping conference, the brain PET community was challenged with a computational contest involving a simulated neurotransmitter release experiment. Fourteen international teams analysed the same imaging dataset, for which the ground-truth was known. Despite a plurality of methods, the solutions were consistent across participants, although not identical. These results should create awareness that the increased sharing of PET data alone will only be one component of enhancing confidence in neuroimaging results and that it will be important to complement this with full details of the analysis pipelines and procedures that have been used to quantify data.
BibTeX
@article{veronese2021reproducibility,
  author = {Mattia Veronese and Gaia Rizzo and Martín A. Belzunce and Julia Schubert and Graham E. Searle and Alex Whittington and Ayla Mansur and Joel Dunn and Andrew J. Reader and Roger N. Gunn and and the Grand Challenge Participants},
  title = {{Reproducibility of findings in modern PET neuroimaging: insight from the NRM2018 grand challenge}},
  journal = {Journal of Cerebral Blood Flow \& Metabolism},
  year = {2021},
  doi = {10.1177/0271678x211015101},
}
Figure