Masked Co-attentional Transformer reconstructs 100x ultra-fast/low-dose whole-body PET from longitudinal images and anatomically guided MRI

Yan-Ran Wang, Liangqiong Qu, Natasha Diba Sheybani, Xiaolong Luo, Jiangshan Wang, Kristina Elizabeth Hawk, Ashok Joseph Theruvath, Sergios Gatidis, Xuerong Xiao, Allison Pribnow, Daniel L. Rubin, Heike E. Daldrup-LinkarXiv 2022 · OpenReview

TL;DR

This paper introduces Masked-LMCTrans, a novel deep learning model that reconstructs whole-body PET scans at 100x lower radiation doses by leveraging longitudinal PET/MRI data. This breakthrough significantly reduces radiation exposure for patients, particularly children, while maintaining high image quality, paving the way for safer and more frequent diagnostic imaging.

What Problem Does This Paper Address?

Whole-body PET scans are crucial for cancer diagnosis and monitoring, but they come with significant radiation exposure and can be time-consuming. For pediatric patients, who often require multiple scans, this radiation burden is a major concern. Existing methods for reducing PET dose or scan time often lead to a substantial degradation in image quality. This paper tackles the challenge of achieving ultra-low-dose (1% of standard) whole-body PET reconstruction without sacrificing diagnostic accuracy, a feat not previously accomplished.

Methods

The core innovation is the Masked-LMCTrans model, a longitudinal multi-modality co-attentional CNN-Transformer. It ingeniously uses paired baseline and follow-up PET/MRI scans from the same patient. The model employs:

  • Dual Encoding Streams: Separate streams process the baseline and follow-up PET/MRI data.
  • DenseNet Encoder: This feature extractor efficiently captures complex patterns from both PET and MRI modalities.
  • Co-attentional Transformer Layers: This is the key to longitudinal fusion. These layers allow the model to learn the global similarity and temporal dependencies between the baseline and follow-up scans, enabling the follow-up scan reconstruction to benefit from the higher quality baseline information.
  • Masking of Tumor Regions: Crucially, the tumor areas in the baseline PET scans are masked out. This prevents the model from introducing false positive "upstaging" in the reconstructed follow-up scans, which is important as tumors often shrink or resolve after treatment.
  • False Focal Loss (FF Loss): A novel loss function designed to penalize incorrect hyperintense pixels, specifically addressing the issue of false positive errors in ultra-low-dose PET reconstruction.

Datasets

  • Stanford PET/MRI Cohort: A primary cohort of paired pre-treatment and post-treatment whole-body PET/MRI scans from pediatric lymphoma patients was used for training and testing.
  • Tübingen PET/MRI Cohort: An independent, cross-continental validation cohort from Tübingen University, consisting of paired baseline and follow-up PET/MRI scans from pediatric lymphoma patients, was used to assess the model's generalizability.

Key Insights and Ideas

  • Longitudinal Data is Key: The paper strongly advocates for the under-explored potential of using longitudinal (time-series) medical imaging data for reconstruction tasks. The inherent redundancy across scans provides valuable context.
  • Co-attention for Fusion: The co-attentional transformer mechanism is highly effective at integrating information from different time points and modalities, going beyond simple concatenation.
  • Masking for Clinical Relevance: Masking tumor regions in the baseline scan is a clever trick to ensure the reconstructed images accurately reflect treatment response, avoiding misleading "upstaging."
  • Ultra-low-dose PET is Achievable: The successful reconstruction of 100x ultra-low-dose PET scans demonstrates that significant radiation reduction is feasible without catastrophic loss of image quality.

Key Results

  • High Image Quality: Masked-LMCTrans reconstructed 1% ultra-low-dose PET images with significantly less noise and more detailed structures compared to simulated 1% PET and even other advanced models.
  • Quantitative Improvements: The model achieved substantial improvements in SSIM (Structural Similarity Index), PSNR (Peak Signal-to-Noise Ratio), and VIF (Visual Information Fidelity) metrics compared to baseline models and simulated low-dose PET. For example, on the Stanford cohort, SSIM improved by ~15.8%, PSNR by ~23.4%, and VIF by ~186% compared to the 1% low-dose PET.
  • Generalizability: The model demonstrated strong performance on the independent Tübingen validation cohort, indicating its robustness across different institutions and scanners.
  • Clinical Impact: The reconstructed images are comparable in quality to standard-dose PET scans, suggesting that this approach can enable safer imaging for pediatric patients and facilitate more frequent monitoring.

Limitations and Future Work

  • Robustness to Lesion Variability: The study acknowledges that lymphoma typically has high metabolic activity. Further work is needed to assess performance with less robust or more subtle lesions, and in other tissue types like bone.
  • Data Demand: Deep learning models inherently require large datasets. The authors note that the lack of large-scale, publicly available PET/MRI datasets is a bottleneck for further scaling.
  • Modality Generalization: While the paper suggests the framework could be adapted for CT and contrast-enhanced MRI, this was not explored in the current work. Future research could investigate its application to other imaging modalities.

Summary updated

Abstract

Despite its tremendous value for the diagnosis, treatment monitoring and surveillance of children with cancer, whole body staging with positron emission tomography (PET) is time consuming and associated with considerable radiation exposure. 100x (1% of the standard clinical dosage) ultra-low-dose/ultra-fast whole-body PET reconstruction has the potential for cancer imaging with unprecedented speed and improved safety, but it cannot be achieved by the naive use of machine learning techniques. In this study, we utilize the global similarity between baseline and follow-up PET and magnetic resonance (MR) images to develop Masked-LMCTrans, a longitudinal multi-modality co-attentional CNN-Transformer that provides interaction and joint reasoning between serial PET/MRs of the same patient. We mask the tumor area in the referenced baseline PET and reconstruct the follow-up PET scans. In this manner, Masked-LMCTrans reconstructs 100x almost-zero radio-exposure whole-body PET that was not possible before. The technique also opens a new pathway for longitudinal radiology imaging reconstruction, a significantly under-explored area to date. Our model was trained and tested with Stanford PET/MRI scans of pediatric lymphoma patients and evaluated externally on PET/MRI images from T\"ubingen University. The high image quality of the reconstructed 100x whole-body PET images resulting from the application of Masked-LMCTrans will substantially advance the development of safer imaging approaches and shorter exam-durations for pediatric patients, as well as expand the possibilities for frequent longitudinal monitoring of these patients by PET.
BibTeX
@misc{wang2022masked,
  author = {Yan-Ran Wang and Liangqiong Qu and Natasha Diba Sheybani and Xiaolong Luo and Jiangshan Wang and Kristina Elizabeth Hawk and Ashok Joseph Theruvath and Sergios Gatidis and Xuerong Xiao and Allison Pribnow and Daniel L. Rubin and Heike E. Daldrup-Link},
  title = {{Masked Co-attentional Transformer reconstructs 100x ultra-fast/low-dose whole-body PET from longitudinal images and anatomically guided MRI}},
  year = {2022},
  eprint = {2205.04044},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.04044},
}
Figure