Recurrent Variational Network: A Deep Learning Inverse Problem Solver applied to the task of Accelerated MRI Reconstruction

George Yiasemis, Jan-Jakob Sonke, Clarisa Sánchez, Jonas Teuwen

Abstract

Magnetic Resonance Imaging (MRI) produces detailed anatomical images but suffers from long acquisition times, leading to motion artifacts and limited throughput. Accelerated MRI techniques reconstruct images from sub-sampled k-space data. We present the Recurrent Variational Network (RecurrentVarNet), a novel deep learning-based inverse problem solver. RecurrentVarNet uses unrolled optimization with recurrent blocks that operate directly in the k-space domain. Each block refines the observed k-space using a data consistency term and a recurrent unit with a learned hidden state. Our method achieves state-of-the-art results on public brain datasets, outperforming existing approaches in both qualitative and quantitative metrics.

Introduction: The Challenge of Accelerated MRI

MRI is a vital diagnostic tool, yet its slow acquisition speed—often requiring patients to remain still for 30–45 minutes—is a significant drawback. This slowness increases costs and makes scans susceptible to motion artifacts. To address this, "Accelerated MRI" sub-samples the frequency domain (k-space), intentionally skipping data points to reduce scan time.

Reconstructing a high-fidelity image from this sparse data is an ill-posed inverse problem. While traditional methods like Compressed Sensing provide a baseline, they often struggle with artifacts at high acceleration factors. Recent deep learning models have shown promise, but many fail to fully exploit the raw k-space data or lack the capacity to pass complex information effectively between iterative reconstruction steps.

Methodology: The RecurrentVarNet Architecture

The RecurrentVarNet treats the reconstruction task as an "unrolled" optimization process, utilizing a sequence of recurrent blocks to iteratively refine the image.

Detailed architecture of the RecurrentVarNet, showing the Sensitivity Estimation - Refinement (SER) module, the Recurrent State Initializer (RSI), and the internal structure of the recurrent blocks operating in k-space.

Key Technical Components

  • Recurrent Optimization in K-Space: Unlike image-domain models, RecurrentVarNet performs updates in the k-space domain. This ensures that the reconstructed output remains consistent with the actual physical measurements from the scanner.
  • Convolutional GRUs (ConvGRUs): The network maintains a "memory" or hidden state ($h_t$) that is passed between blocks. This allows the model to learn and preserve complex anatomical features throughout the optimization process.
  • Sensitivity Estimation and Refinement (SER): MRI uses multiple receiver coils. RecurrentVarNet includes an integrated U-Net that learns to refine coil sensitivity maps ($\\mathbf{S}$) on the fly, improving upon static, pre-calculated maps.
  • Recurrent State Initializer (RSI): This module analyzes the initial sub-sampled input to generate the starting hidden state ($h_0$), providing a "warm start" for the recurrent process.

By combining these elements, the model creates a hybrid domain learning environment: it uses recurrent units to process image features while projecting results back into k-space for data consistency checks.

Results and Evaluation

The model was evaluated on the Calgary-Campinas public brain dataset (1.5T and 3T) and the fastMRI AXT1 multi-coil brain dataset. Tests were conducted at 5-fold and 10-fold acceleration (using only 20% and 10% of the data, respectively).

Reconstructed brain MRI slices demonstrating the high-fidelity output achieved by the RecurrentVarNet.

Quantitative Performance

On the Calgary-Campinas dataset at 5x acceleration, RecurrentVarNet achieved a Structural Similarity Index (SSIM) of 0.9418. This outperformed other leading models such as XPDNet (0.9325) and the Recurrent Inference Machine (0.9381).

Qualitative Fidelity

Visual analysis confirmed that RecurrentVarNet excels at preserving fine structural details and minimizing noise, particularly at extreme 10x acceleration where other models typically exhibit significant blurring or hallucinatory artifacts.

Conclusion and Future Directions

RecurrentVarNet demonstrates that treating MRI reconstruction as a recurrent sequence is superior to standard feed-forward architectures. The use of hidden states allows for a richer anatomical representation, while k-space consistency prevents the network from deviating from the raw data.

A primary limitation is the memory cost associated with training many iterations ($T$) and large hidden states ($n_l$). Future research will focus on improving memory efficiency and extending the architecture to dynamic 4D MRI, where temporal recurrence could provide even greater benefits for capturing motion.

Code is available at: https://github.com/NKI-AI/direct

Summary updated

Abstract

Magnetic Resonance Imaging can produce detailed images of the anatomy and physiology of the human body that can assist doctors in diagnosing and treating pathologies such as tumours. However, MRI suffers from very long acquisition times that make it susceptible to patient motion artifacts and limit its potential to deliver dynamic treatments. Conventional approaches such as Parallel Imaging and Compressed Sensing allow for an increase in MRI acquisition speed by reconstructing MR images from sub-sampled MRI data acquired using multiple receiver coils. Recent advancements in Deep Learning combined with Parallel Imaging and Compressed Sensing techniques have the potential to produce high-fidelity reconstructions from highly accelerated MRI data. In this work we present a novel Deep Learning-based Inverse Problem solver applied to the task of Accelerated MRI Reconstruction, called the Recurrent Variational Network (RecurrentVarNet), by exploiting the properties of Convolutional Recurrent Neural Networks and unrolled algorithms for solving Inverse Problems. The RecurrentVarNet consists of multiple recurrent blocks, each responsible for one iteration of the unrolled variational optimization scheme for solving the inverse problem of multi-coil Accelerated MRI Reconstruction. Contrary to traditional approaches, the optimization steps are performed in the observation domain (k-space) instead of the image domain. Each block of the RecurrentVarNet refines the observed k-space and comprises a data consistency term and a recurrent unit which takes as input a learned hidden state and the prediction of the previous block. Our proposed method achieves new state of the art qualitative and quantitative reconstruction results on 5-fold and 10-fold accelerated data from a public multi-coil brain dataset, outperforming previous conventional and deep learning-based approaches. Our code is publicly available at https://github.com/NKI-AI/direct.
BibTeX
@inproceedings{yiasemis2022recurrent,
  author = {George Yiasemis and Jan-Jakob Sonke and Clarisa Sánchez and Jonas Teuwen},
  title = {{Recurrent Variational Network: A Deep Learning Inverse Problem Solver applied to the task of Accelerated MRI Reconstruction}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  url = {https://openreview.net/forum?id=yBmOMLrSwF},
}
Figure