Memory-Efficient Training for Fully Unrolled Deep Learned PET Image Reconstruction With Iteration-Dependent Targets

Guillaume Corda-D’Incan, Julia A. Schnabel, Andrew J. ReaderIEEE Transactions on Radiation and Plasma Medical Sciences 2021 · DOI

TL;DR

This paper introduces a more memory-efficient way to train deep learning models for PET image reconstruction, called FBSEM-IS-Net. By using iteration-dependent targets and a sequential training approach, they enable the training of much deeper networks, significantly reducing memory requirements (up to 98%) without sacrificing performance.

What Problem Does This Paper Address?

Positron Emission Tomography (PET) is crucial for medical imaging, but the data it collects is inherently noisy due to physical processes and the need to minimize radiation exposure. Traditional reconstruction methods like MLEM/OSEM struggle with this noise, leading to "night sky" images. While regularization techniques and model-based approaches (like FBSEM-Net) have improved image quality, training deep, "unrolled" versions of these networks, especially for 3D data, is computationally prohibitive due to massive memory demands. This is often addressed by using "leapfrogging," where fewer network modules are trained than the target reconstruction iterations, potentially limiting performance and generalization.

Methods

The authors propose a new version of the Forward-Backward Splitting Expectation-Maximization Network (FBSEM-Net), termed FBSEM-IS-Net, with three key modifications:

  1. Iteration-Dependent Networks: The regularisation network's parameters are adapted for each iteration of the reconstruction process, allowing for customized regularization as noise levels change.
  2. Iteration-Dependent Targets and Losses: Instead of a single final target, each module in the unrolled network is trained to match the corresponding iteration's output from a high-quality, noise-free reconstruction. This constrains the network to act as a pure denoiser.
  3. Sequential Training: This is the core memory-saving innovation. Instead of training all network modules simultaneously (end-to-end), each module is trained independently and sequentially. This drastically reduces memory usage by avoiding the need to store all intermediate activations for backpropagation.

Datasets

The study utilizes simulated PET data for both 2D and 3D scenarios.

  • 2D Datasets: Twenty brain phantoms from BrainWeb were used to simulate 2D [18F]FDG PET images. High-count reconstructions (100M) served as targets, while low-count (500k) data was simulated with Poisson noise. 80 samples were used for training, 10 for validation, and 10 for testing.
  • 3D Datasets: Realistic 3D brain PET-MR phantoms were generated from T1-weighted MPRAGE MR images of 11 patients. These phantoms incorporated anatomical information, simulated uptake values, and spherical lesions. Noisy sinograms were generated with PSF modeling, attenuation, normalization, and Poisson noise. The training set comprised 45 samples, with 5 samples each for validation and testing.

Key Insights and Ideas

  • Iteration-dependent targets are crucial for generalization: By forcing the network to match intermediate targets, the authors constrain the learned regularizers to act purely as denoisers, preventing them from learning image details and thus improving generalization.
  • Sequential training is a game-changer for memory: The proposed sequential training method dramatically reduces memory requirements (up to 98%), making it feasible to train very deep unrolled networks (e.g., 60 modules for 3D data) on standard GPUs, which was previously impossible.
  • Truncated backpropagation doesn't hurt performance: Surprisingly, even with sequential training, which effectively truncates backpropagation, the network's performance is not significantly impacted compared to conventional end-to-end training.
  • Deeper networks are better: The ability to train deeper networks (more modules) with FBSEM-IS-Net leads to improved reconstruction quality, particularly in preserving lesion shapes.

Key Results

  • Improved Reconstruction Quality: FBSEM-IS-Net consistently outperformed other methods, including the original FBSEM-Net and INN variants, in terms of Normalized Root Mean Square Error (NRMSE) on both 2D and 3D simulated data. Qualitatively, it produced sharper images with better preservation of lesion shapes.
  • Memory Reduction: Sequential training reduced memory usage from 222.0 GB (original FBSEM-Net) and 224.0 GB (FBSEM-IS-Net end-to-end) to just 3.7 GB for FBSEM-IS-Net with sequential training.
  • Robustness: FBSEM-IS-Net demonstrated good generalisation abilities across different radiotracer distributions and noise levels.
  • Consistency: Training at the module-level with iteration-dependent targets improved the consistency of network parameter optimization across multiple training runs.

Limitations and Future Work

  • Generalization Assessment: While FBSEM-IS-Net showed good generalization, the authors suggest that a more comprehensive assessment across a wider range of distributions is still needed.
  • Loss Function Exploration: The current MSE loss might contribute to affected lesion shapes. The authors propose exploring other loss functions, such as Structural Similarity (SSIM), to potentially improve lesion shape and uptake reconstruction.
  • Theoretical Investigation of Sequential Training: The impact of removing some backpropagation in sequential training warrants theoretical investigation.
  • Application to Other Unrolled Methods: The proposed iteration-dependent targets/losses and sequential training are demonstrated for FBSEM-Net but are suggested to be applicable to any unrolled deep learning method.

Summary updated

Abstract

We propose a new version of the forward-backward splitting expectation-maximisation network (FBSEM-Net) along with a new memory-efficient training method enabling the training of fully unrolled implementations of 3D FBSEM-Net. FBSEM-Net unfolds the maximum <i>a posteriori</i> expectation-maximisation algorithm and replaces the regularisation step by a residual convolutional neural network. Both the gradient of the prior and the regularisation strength are learned from training data. In this new implementation, three modifications of the original framework are included. First, iteration-dependent networks are used to have a customised regularisation at each iteration. Second, iteration-dependent targets and losses are introduced so that the regularised reconstruction matches the reconstruction of noise-free data at every iteration. Third, sequential training is performed, making training of large unrolled networks far more memory efficient and feasible. Since sequential training permits unrolling a high number of iterations, there is no need for artificial use of the regularisation step as a leapfrogging acceleration. The results obtained on 2D and 3D simulated data show that FBSEM-Net using iteration-dependent targets and losses improves the consistency in the optimisation of the network parameters over different training runs. We also found that using iteration-dependent targets increases the generalisation capabilities of the network. Furthermore, unrolled networks using iteration-dependent regularisation allowed a slight reduction in reconstruction error compared to using a fixed regularisation network at each iteration. Finally, we demonstrate that sequential training successfully addresses potentially serious memory issues during the training of deep unrolled networks. In particular, it enables the training of 3D fully unrolled FBSEM-Net, not previously feasible, by reducing the memory usage by up to 98% compared to a conventional end-to-end training. We also note that the truncation of the backpropagation (due to sequential training) does not notably impact the network's performance compared to conventional training with a full backpropagation through the entire network.
BibTeX
@article{cordadincan2021memory,
  author = {Guillaume Corda-D’Incan and Julia A. Schnabel and Andrew J. Reader},
  title = {{Memory-Efficient Training for Fully Unrolled Deep Learned PET Image Reconstruction With Iteration-Dependent Targets}},
  journal = {IEEE Transactions on Radiation and Plasma Medical Sciences},
  year = {2021},
  doi = {10.1109/trpms.2021.3101947},
}
Figure