Physics-Driven Autoregressive State Space Models for Medical Image Reconstruction

Bilal Kabas, Fuat Arslan, Valiyeh Ansarian Nezhad, Saban Öztürk, Emine Ulku Saritas, Tolga ÇukurarXiv 2024 · DOI · arXiv

Introduction

Medical imaging modalities like MRI and CT are essential for diagnosis but often require long acquisition times or involve radiation exposure. To address these issues, undersampling is frequently used, but reconstructing high-fidelity images from incomplete data is an ill-posed inverse problem. While Convolutional Neural Networks (CNNs) and Transformers have been applied to this task, they often face a trade-off between local receptive fields and high computational costs. This paper introduces MambaRoll, a physics-driven reconstruction framework that integrates State Space Models (SSMs) with a multi-scale autoregressive architecture to achieve superior performance and efficiency.

Methodology

MambaRoll is structured as a series of physics-driven unrolling cascades. Each cascade alternates between a learned neural network prior and a Data Consistency (DC) layer, ensuring the reconstruction remains faithful to physical measurements (k-space in MRI or Radon transform in CT).

The MambaRoll architecture utilizes a series of cascades with multi-scale autoregressive linking to recover fine anatomical details from global structures.

A key innovation is the Multi-Scale Autoregressive (AR) Linking. Within each cascade, the model predicts finer-scale feature maps conditioned on coarser-scale representations. This top-down flow ensures that global anatomical structures guide the recovery of fine details. To handle high-resolution 2D images efficiently, the framework utilizes Compressed PD-SSM Modules, which employ space-to-depth transformations to shorten sequence lengths for the Mamba blocks, allowing for linear complexity relative to image size.

Experimental Results

The model was evaluated on the fastMRI (Brain) and LoDoPaB-CT datasets. MambaRoll consistently outperformed state-of-the-art CNN and Transformer baselines in both image quality and computational efficiency.

Comparison of PSNR vs. Latency for MRI and CT reconstruction. MambaRoll achieves superior image quality (PSNR) while maintaining competitive inference speeds and lower memory footprints.

In MRI reconstruction, MambaRoll achieved an average PSNR of 40.91 dB, significantly outperforming the physics-driven Transformer baseline (DSFormer, 39.57 dB) and the standard PD-CNN (MoDL, 38.39 dB). Notably, MambaRoll is nearly 8x faster than DSFormer while using less memory. In sparse-view CT, the model offered a ~3 dB gain over Transformer-based methods and a ~4.8 dB gain over CNN-based methods. Visualizations of the Effective Receptive Field (ERF) further confirmed that MambaRoll captures a much broader context than its predecessors.

Conclusion

MambaRoll demonstrates that the integration of State Space Models with multi-scale autoregressive linking provides a powerful solution for medical image reconstruction. By maintaining a massive receptive field with linear scaling, it overcomes the limitations of windowed attention in Transformers and local kernels in CNNs. Future work will explore extending this framework to self-supervised learning and multi-modal reconstruction tasks.

Summary updated

BibTeX
@misc{kabas2024physics,
  author = {Bilal Kabas and Fuat Arslan and Valiyeh Ansarian Nezhad and Saban Öztürk and Emine Ulku Saritas and Tolga Çukur},
  title = {{Physics-Driven Autoregressive State Space Models for Medical Image Reconstruction}},
  year = {2024},
  eprint = {2412.09331},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.09331},
}
Figure