Nuclear Medicine and Artificial Intelligence: Best Practices for Algorithm Development

Tyler Bradshaw, Ronald Boellaard, Joyita Dutta, Abhinav K. Jha, Paul Jacobs, Quanzheng Li, Chi Liu, Arkadiusz Sitek, Babak Saboury, Peter J. H. Scott, Piotr J. Slomka, John J. Sunderland, Richard L. Wahl, Fereshteh Yousefirizi, Sven Zuehlsdorff, Arman Rahmim, Irène BuvatJournal of Nuclear Medicine 2021 · DOI

Abstract

The nuclear medicine field has seen a rapid expansion of academic and commercial interest in developing artificial intelligence (AI) algorithms. Users and developers can avoid some of the pitfalls of AI by recognizing and following best practices in AI algorithm development. In this article, recommendations on technical best practices for developing AI algorithms in nuclear medicine are provided, beginning with general recommendations and then continuing with descriptions of how one might practice these principles for specific topics within nuclear medicine. This report was produced by the AI Task Force of the Society of Nuclear Medicine and Molecular Imaging.

Introduction

As publications in nuclear medicine AI continue to grow exponentially, the field faces a critical "reproducibility crisis." This challenge is often fueled by small datasets, inconsistent validation techniques, and the unique complexities of nuclear medicine data, such as high noise levels and lower image volumes compared to other modalities.

Many current studies fall into common traps, reporting overly optimistic performance due to data leakage or failing to generalize across different hospital scanners. To move AI from academic curiosity to a trusted clinical tool, the AI Task Force of the Society of Nuclear Medicine and Molecular Imaging (SNMMI) has established a standardized framework for algorithm development. This framework distinguishes between "proof-of-concept" method development and the rigorous evaluation required for clinical translation.

The 8-Step Development Pipeline

Rather than focusing on a single architecture, the authors propose a structured, eight-step pipeline designed to ensure that AI tools are robust, reproducible, and clinically relevant.

The proposed 8-step pipeline for robust AI algorithm development, spanning from study design to post-deployment evaluation.
  1. Study Design: Defining the clinical task in close collaboration with domain experts.
  2. Data Collection: Anticipating technical, demographic, or selection-based biases.
  3. Data Labeling: Establishing a clear hierarchy of "truth" for the training data.
  4. Model Design: Comparing complex novel models against simpler, established baselines.
  5. Model Training: Implementing rigorous data partitioning to prevent leakage.
  6. Model Testing: Evaluating performance on unseen, ideally external, datasets.
  7. Reporting: Adhering to standardized checklists like MI-CLAIM or SPIRIT-AI.
  8. Evaluation: Monitoring for "performance drift" after the tool is deployed.

Data Labeling and the Hierarchy of Truth

A significant hurdle in nuclear medicine is the lack of standardized "ground truth" labels. The paper emphasizes that not all labels are created equal and introduces a hierarchy of diagnostic annotation quality.

The hierarchy of diagnostic annotation quality, ranking labels from approximate crowd-sourced data to definitive ground truth like biopsy or surgical confirmation.

For clinical tasks, the authors argue that having multiple expert annotators adjudicate labels is often more valuable than having a larger dataset with noisy or "approximate" labels. High-quality labels—such as those confirmed by biopsy, surgery, or high-fidelity simulations—are essential for training models that clinicians can trust.

Technical Best Practices and Pitfalls

To ensure scientific integrity, the task force highlights several key technical considerations:

  • Preventing Data Leakage: A common failure occurs when information from the test set "seeps" into training—for example, by performing feature selection on the entire dataset before splitting. The paper advocates for nested cross-validation to maintain strict separation.
  • Interpretability: For clinical adoption, algorithms must be "explainable." Techniques like Saliency Maps or Shapley values should be used to show physicians the reasoning behind a specific prediction.
  • Ablation Studies: Researchers should systematically remove parts of a model to identify which components truly contribute to its performance.
  • Metrics Matter: Standard computer vision metrics can be misleading. In a medical context, a small error in a diagnostically critical area is far more significant than a large error in the background.

Conclusion and Future Directions

This roadmap serves as a checklist for the field, emphasizing transparency and the sharing of code and models. While institutional data silos and de-identification challenges remain, following these best practices will help bridge the gap between development and clinical utility. Future work from the AI Task Force will focus specifically on the rigorous methods required for prospective clinical validation of these "frozen" algorithms.

Summary updated

Abstract

The nuclear medicine field has seen a rapid expansion of academic and commercial interest in developing artificial intelligence (AI) algorithms. Users and developers can avoid some of the pitfalls of AI by recognizing and following best practices in AI algorithm development. In this article, recommendations on technical best practices for developing AI algorithms in nuclear medicine are provided, beginning with general recommendations and then continuing with descriptions of how one might practice these principles for specific topics within nuclear medicine. This report was produced by the AI Task Force of the Society of Nuclear Medicine and Molecular Imaging.
BibTeX
@article{bradshaw2021nuclear,
  author = {Tyler Bradshaw and Ronald Boellaard and Joyita Dutta and Abhinav K. Jha and Paul Jacobs and Quanzheng Li and Chi Liu and Arkadiusz Sitek and Babak Saboury and Peter J. H. Scott and Piotr J. Slomka and John J. Sunderland and Richard L. Wahl and Fereshteh Yousefirizi and Sven Zuehlsdorff and Arman Rahmim and Irène Buvat},
  title = {{Nuclear Medicine and Artificial Intelligence: Best Practices for Algorithm Development}},
  journal = {Journal of Nuclear Medicine},
  year = {2021},
  doi = {10.2967/jnumed.121.262567},
}
Figure