Spach Transformer: Spatial and Channel-Wise Transformer Based on Local and Global Self-Attentions for PET Image Denoising

Se-In Jang, Tinsu Pan, Ye Li, Pedram Heidari, Junyu Chen, Quanzheng Li, Kuang GongIEEE TMI 2022 · OpenReview

TL;DR

This paper introduces the Spach Transformer, a novel deep learning architecture for denoising PET (Positron Emission Tomography) images. By combining spatial and channel-wise self-attention mechanisms, it effectively reduces noise while preserving crucial details like small tumor uptakes, outperforming existing state-of-the-art methods.

What Problem Does This Paper Address?

Positron Emission Tomography (PET) is a powerful imaging technique for visualizing biological processes, but its images often suffer from low signal-to-noise ratio (SNR). This noise can obscure important details, especially small lesions or tumor uptakes, hindering accurate diagnosis and monitoring. While Convolutional Neural Networks (CNNs) have been successful in denoising, their limited receptive fields struggle to capture long-range dependencies in 3D PET data. Global self-attention mechanisms, like those in Transformers, can capture these long-range interactions but are computationally expensive for 3D images. This work aims to bridge this gap by developing an efficient Transformer-based model that leverages both local and global spatial and channel information for high-quality PET image denoising.

Methods

The core of this work is the Spach Transformer, an encoder-decoder architecture designed for PET image denoising. It uniquely combines two types of self-attention:

  • Spatial-wise Transformer Blocks: These blocks utilize a window-based and shifted window-based multi-head self-attention (MSA) mechanism, inspired by Swin Transformer. This allows for efficient computation of local spatial self-attention, capturing fine-grained spatial relationships.
  • Channel-wise Transformer Blocks: These blocks employ a multi-Dconv head transposed attention (MDTA) module, similar to Restormer. This mechanism computes self-attention along the channel dimension, enabling global context aggregation with linear computational complexity.

To address the potential loss of spatial information in channel-wise attention, the authors introduce a Gated-Conv Feed-Forward Network (GCFN). This module integrates spatial attributes by using 3x3x3 spatial-wise convolutions within the channel-wise transformer block, aiming to improve lesion uptake recovery. The encoder progressively extracts features, while the decoder reconstructs the denoised image, utilizing skip connections to preserve details.

Datasets

The Spach Transformer was evaluated on clinical whole-body PET datasets acquired from a GE DMI PET/CT scanner. The datasets included four types of PET tracers, commonly used in oncology:

  • $^{18}$F-FDG
  • $^{18}$F-ACBC
  • $^{18}$F-DCFPyL
  • $^{68}$Ga-DOTATATE

For training and validation, datasets of $^{18}$F-FDG and $^{18}$F-ACBC were used. The testing set comprised datasets from all four tracers, including $^{18}$F-DCFPyL and $^{68}$Ga-DOTATATE, which were not part of the training set, to evaluate the model's generalization capabilities. Low-dose data was generated by downsampling the original event counts (1/4 and 1/16), and normal-dose images served as ground truth.

Key Insights and Ideas

  • Synergy of Spatial and Channel Attention: The paper demonstrates that effectively combining both spatial and channel-wise self-attention is crucial for PET denoising. Spatial attention excels at preserving local details and structures, while channel attention captures global dependencies, leading to better overall image quality.
  • GCFN for Spatial Preservation: The introduction of the GCFN module is a key innovation. It effectively mitigates the potential loss of spatial information that can occur with purely channel-wise attention, leading to improved recovery of subtle tumor uptakes.
  • Transformer Efficiency for 3D Data: By using window-based spatial attention and transposed channel attention, the Spach Transformer achieves a balance between capturing long-range dependencies and maintaining computational efficiency for 3D PET data, which is a significant challenge for global attention mechanisms.
  • Generalization Across Tracers: The model shows strong performance even on tracers not seen during training, suggesting its ability to learn generalizable features for PET denoising.

Key Results

The Spach Transformer consistently outperformed state-of-the-art methods, including Unet, Swin Transformer, and Restormer, across various metrics:

  • Quantitative Performance: On 1/4 low-dose datasets, Spach Transformer achieved an average PSNR of 55.0840 and an average SSIM of 0.9404. On 1/16 low-dose datasets, it achieved an average PSNR of 54.0678 and an average SSIM of 0.9176. These results were statistically significant.
  • Tumor Uptake Recovery: The Spach Transformer demonstrated superior performance in preserving and highlighting tumor uptakes, as evidenced by higher Contrast-to-Noise Ratio (CNR) values compared to other methods.
  • Visual Quality: Qualitative results (Figs. 5, 7-10, 15-18) show that the Spach Transformer produces denoised images with significantly less noise and better preservation of fine details and lesion uptakes compared to the reference methods.
  • Model Efficiency: The Spach Transformer achieved these results with fewer model parameters (19M) compared to some of the reference methods (e.g., 21M for Unet, Swin Transformer, and Restormer).

Limitations and Future Work

  • Model Size vs. SSIM: The authors noted that while PSNR generally increased with model size, SSIM sometimes decreased. They suggest that the best model was chosen based on PSNR for denoising, and future work could explore loss functions that more directly incorporate SSIM.
  • Loss Functions: The paper primarily focused on image quality metrics like PSNR and SSIM. A potential future research direction is to incorporate loss functions that consider other local features, such as radiomics of lesions, to further enhance performance.
  • Intra-tumor Heterogeneity: The authors suggest that future research could focus on detecting lesions with low activity to better evaluate the impact of intra-tumor heterogeneity.
  • Clinical Validation: While evaluated on clinical datasets, further extensive evaluations using diverse clinical datasets are planned.

Summary updated

Abstract

Position emission tomography (PET) is widely used in clinics and research due to its quantitative merits and high sensitivity, but suffers from low signal-to-noise ratio (SNR). Recently convolutional neural networks (CNNs) have been widely used to improve PET image quality. Though successful and efficient in local feature extraction, CNN cannot capture long-range dependencies well due to its limited receptive field. Global multi-head self-attention (MSA) is a popular approach to capture long-range information. However, the calculation of global MSA for 3D images has high computational costs. In this work, we proposed an efficient spatial and channel-wise encoder-decoder transformer, Spach Transformer, that can leverage spatial and channel information based on local and global MSAs. Experiments based on datasets of different PET tracers, i.e., 18F-FDG, 18F-ACBC, 18F-DCFPyL, and 68Ga-DOTATATE, were conducted to evaluate the proposed framework. Quantitative results show that the proposed Spach Transformer framework outperforms state-of-the-art deep learning architectures.
BibTeX
@article{jang2022spach,
  author = {Se-In Jang and Tinsu Pan and Ye Li and Pedram Heidari and Junyu Chen and Quanzheng Li and Kuang Gong},
  title = {{Spach Transformer: Spatial and Channel-Wise Transformer Based on Local and Global Self-Attentions for PET Image Denoising}},
  journal = {IEEE Trans. Medical Imaging},
  year = {2022},
  url = {https://openreview.net/forum?id=iBnOjUy1rs},
}
Figure