Computationally Efficient 3D MRI Reconstruction with Adaptive MLP

Eric Z. Chen, Chi Zhang, Xiao Chen, Yikang Liu, Terrence Chen, Shanhui Sun

TL;DR

This paper introduces Recon3DMLP, a novel hybrid deep learning architecture for accelerating 3D MRI reconstruction. By combining the strengths of Convolutional Neural Networks (CNNs) for local feature extraction and adaptive Multi-Layer Perceptrons (MLPs) for global context, Recon3DMLP significantly improves reconstruction quality and efficiency, addressing key limitations of existing methods.

What Problem Does This Paper Address?

Reconstructing high-resolution (HR) 3D MRI images is crucial for detailed anatomical analysis, but it's computationally demanding. Traditional methods often rely on CNNs with small kernels, which struggle to capture the long-range dependencies required for deconvolution tasks inherent in MRI reconstruction. This limitation is exacerbated by the massive data sizes of 3D MRI, leading to significant GPU memory constraints that hinder the use of larger, more powerful models. Existing solutions often involve trade-offs between reconstruction quality, speed, and memory usage, making it challenging to deploy efficient and effective 3D MRI reconstruction in clinical settings.

Methods

The core innovation is the Recon3DMLP architecture, a cascaded network that strategically combines CNNs and adaptive MLPs (dMLPs).

  • Hybrid CNN-dMLP Cascades: Each cascade in the network starts with CNN modules using small kernels (e.g., k=3) to efficiently reconstruct low-frequency information. These are followed by adaptive MLP (dMLP) modules with larger effective kernels to capture high-frequency details and global context. This hybrid approach aims to leverage the strengths of both architectures.
  • Adaptive MLP (dMLP) with Circular Shift: To overcome the fixed input size limitation of standard MLPs and to effectively capture global information from the entire Field of View (FOV), the authors introduce an adaptive MLP module. This module utilizes a "circular padding" and "circular shift" operation, inspired by MRI physics and DFT periodicity. This allows the dMLP to process arbitrary image sizes and extract global information by exchanging information between adjacent patches.
  • Memory-Efficient Data Fidelity (eDF) Module: A novel data fidelity module is proposed to significantly reduce GPU memory consumption. Instead of broadcasting the entire coil-combined image, the eDF module processes data coil-by-coil, leading to a memory reduction of over 50%.
  • Additional Memory-Saving Techniques: The authors also employ standard techniques like gradient checkpointing, RO cropping, and half-precision (FP16) inference to further optimize memory usage.

Datasets

The paper uses a multi-contrast HR 3D brain MRI dataset.

  • Dataset Size: 751 3D multi-coil images for training, 32 for validation, and 29 for testing.
  • Image Dimensions: Ranging from 224×220×96×12 to 336×336×192×32 (RO×PE×SPE×Coil).
  • Benchmark: The dataset is described as a high-resolution (HR) 3D multi-coil MRI dataset, implying it serves as a benchmark for evaluating 3D MRI reconstruction methods.

Key Insights and Ideas

  • MLPs for Global Context in MRI: The paper highlights that MLPs, with their inherent ability to model long-range dependencies, are well-suited for capturing the global information needed in MRI reconstruction, which is often missed by CNNs with small kernels.
  • Adaptive MLPs for Flexibility: The introduction of an adaptive MLP module that can handle arbitrary input sizes, enabled by circular padding and shifting, is a key breakthrough for applying MLPs to real-world, variable-sized medical imaging data.
  • Hybrid Architecture Synergy: The combination of CNNs for local features and MLPs for global context within a cascaded structure proves more effective than using either alone, leading to improved reconstruction quality.
  • Memory Efficiency is Crucial: The development of the eDF module demonstrates that significant memory savings can be achieved through architectural design, not just by relying on general optimization techniques.

Key Results

  • Improved Reconstruction Quality: Recon3DMLP achieves better PSNR and SSIM compared to various CNN-based models, including larger and more complex ones, under similar GPU memory constraints. For instance, compared to a larger Recon3DCNN (e=24), Recon3DMLP uses 24% less GPU memory while achieving comparable or better PSNR/SSIM.
  • Significant Memory Reduction: The eDF module, combined with FP16, reduces inference GPU memory by up to 71% (from >40GB to 11.5GB), making it feasible for mid-range GPUs.
  • Efficient Inference: While the dMLP adds some computational cost, the overall inference time remains competitive, and the memory savings are substantial.
  • Ablation Studies Confirm Effectiveness: Removing shift operations from the dMLP leads to a performance drop, indicating the importance of the global information exchange facilitated by these shifts. Replacing dMLPs with large or small kernel convolutions also results in worse performance, underscoring the advantage of the MLP approach.

Limitations and Future Work

  • Fixed Patch and Shift Size: The current dMLP module uses a fixed patch and shift size. The authors suggest that utilizing multi-scale patch and shift sizes could further enhance performance.
  • Data Dependency: The paper notes that MLP-based models, including Recon3DMLP, may perform poorly if the training dataset is small.
  • Future Work: Investigating multi-scale patch and shift sizes for the dMLP module is a clear direction for future research.

Summary updated

Abstract

Compared with 2D MRI, 3D MRI provides superior volumetric spatial resolution and signal-to-noise ratio. However, it is more challenging to reconstruct 3D MRI images. Current methods are mainly based on convolutional neural networks (CNN) with small kernels, which are difficult to scale up to have sufficient fitting power for 3D MRI reconstruction due to the large image size and GPU memory constraint. Furthermore, MRI reconstruction is a deconvolution problem, which demands long-distance information that is difficult to capture by CNNs with small convolution kernels. The multi-layer perceptron (MLP) can model such long-distance information, but it requires a fixed input size. In this paper, we proposed Recon3DMLP, a hybrid of CNN modules with small kernels for low-frequency reconstruction and adaptive MLP (dMLP) modules with large kernels to boost the high-frequency reconstruction, for 3D MRI reconstruction. We further utilized the circular shift operation based on MRI physics such that dMLP accepts arbitrary image size and can extract global information from the entire FOV. We also propose a GPU memory efficient data fidelity module that can reduce >50\(\%\) memory. We compared Recon3DMLP with other CNN-based models on a high-resolution (HR) 3D MRI dataset. Recon3DMLP improves HR 3D reconstruction and outperforms several existing CNN-based models under similar GPU memory consumption, which demonstrates that Recon3DMLP is a practical solution for HR 3D MRI reconstruction.
BibTeX
@inproceedings{chen2023computationally,
  author = {Eric Z. Chen and Chi Zhang and Xiao Chen and Yikang Liu and Terrence Chen and Shanhui Sun},
  title = {{Computationally Efficient 3D MRI Reconstruction with Adaptive MLP}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2023},
  url = {https://openreview.net/forum?id=6P5bhGz9lC},
}
Figure