Σ-net: Ensembled Iterative Deep Neural Networks for Accelerated Parallel MR Image Reconstruction

Jo Schlemper, Chen Qin, Jinming Duan, Ronald M. Summers, Kerstin HammernikarXiv 2019 · OpenReview

TL;DR

This paper introduces $\Sigma$-net, an ensemble of deep neural networks designed to accelerate Magnetic Resonance Imaging (MRI) by reconstructing images from undersampled k-space data. By combining different network architectures and training strategies, $\Sigma$-net achieves robust and high-quality reconstructions, outperforming individual models and addressing the trade-off between quantitative accuracy and visual texture.

What Problem Does This Paper Address?

Accelerating MRI acquisition is crucial for reducing scan times, improving patient comfort, and minimizing motion artifacts. Parallel MRI (pMRI) techniques, which use multiple receiver coils, are a key enabler of this acceleration. However, acquiring less k-space data (undersampling) leads to aliasing artifacts that need to be removed. Traditional reconstruction methods can be slow or may not fully recover image quality. Deep learning offers a promising avenue for faster and more accurate reconstruction, but challenges remain in achieving both quantitative precision and visually appealing results, especially when adapting to new datasets. This paper tackles the problem of reconstructing high-quality MR images from undersampled data by exploring and ensembling various deep learning reconstruction strategies.

Methods

The core of $\Sigma$-net is built upon an "unrolled optimization" framework, mimicking iterative reconstruction algorithms but with learnable components. The authors explore two main network architectures:

  • Parallel Coil Networks (PCNs): These networks implicitly learn how to weight the signals from individual coils, effectively performing coil combination.
  • Sensitivity Networks (SNs): These networks leverage explicit coil sensitivity maps to reconstruct individual coil images, which are then combined.

The networks are trained using a combination of supervised learning (with content and Generative Adversarial Network - GAN - losses) and various data consistency (DC) approaches, including gradient descent, proximal mappings, and variable splitting. A key innovation is the ensembling of multiple trained models (PCNs and different SN variants) to form the final $\Sigma$-net. This ensembling aims to reduce random errors and improve robustness. Additionally, a style transfer layer (STL) is introduced specifically for the challenge to bridge the visual gap between sensitivity-weighted images and the standard root-sum-of-squares (RSS) reconstruction, although it's noted to decrease quantitative metrics. A semi-supervised finetuning strategy is also explored to adapt models to test-time k-space data, which improves visual texture but can slightly degrade quantitative scores.

Datasets

The primary dataset used is fastMRI [14], a publicly available dataset for accelerated parallel MRI. The authors trained and validated their models on this dataset, specifically using data for acceleration factors R=4 and R=8, and for contrasts PD (proton density) and PDFS (proton density fat-suppressed). Sensitivity maps were estimated from auto-calibration lines (ACLs) within the fastMRI data.

Key Insights and Ideas

  • Ensembling is Key for Robustness: Combining multiple diverse reconstruction models (PCNs and SNs with different DC methods) significantly improves robustness and quantitative scores, effectively averaging out individual model weaknesses.
  • Semi-supervised Finetuning for Texture: While supervised training focuses on quantitative metrics, semi-supervised finetuning on k-space data at test time can yield visually more textured and sharper images, albeit at a slight cost to quantitative performance. This highlights a common trade-off in image reconstruction.
  • STL for Visual Bridging: The Style Transfer Layer (STL) is an interesting, albeit challenge-specific, approach to align the visual appearance of SN reconstructions with RSS targets, addressing perceived visual discrepancies.
  • Quantitative vs. Visual Perception: The paper explicitly notes that even when quantitative metrics are high, visually appealing reconstructions (like those from semi-supervised finetuning) might not always align with the best quantitative scores, underscoring the complexity of evaluating image reconstruction.

Key Results

The ensembled $\Sigma$-net achieved the best quantitative scores (highest SSIM) on the fastMRI validation set for both R=4 and R=8 acceleration factors. For example, at R=8, the $\Sigma$-net achieved an SSIM of 0.8917 ± 0.1317, outperforming individual models. Qualitatively, while the semi-supervised finetuned GD-SN-FT model produced the most textured and sharp images, the $\Sigma$-net provided a good balance of quantitative accuracy and visual quality.

Limitations and Future Work

The authors acknowledge that the semi-supervised finetuning, while visually appealing, decreases quantitative metrics. The Style Transfer Layer (STL) was introduced specifically for the challenge to match RSS contrast and is noted to have no practical relevance beyond that, potentially decreasing visual quality. Future work will focus on evaluating SNs on sensitivity-combined reference images, suggesting a shift towards more perceptually aligned evaluation metrics or reconstruction targets.

Summary updated

Abstract

We explore an ensembled $\Sigma$-net for fast parallel MR imaging, including parallel coil networks, which perform implicit coil weighting, and sensitivity networks, involving explicit sensitivity maps. The networks in $\Sigma$-net are trained in a supervised way, including content and GAN losses, and with various ways of data consistency, i.e., proximal mappings, gradient descent and variable splitting. A semi-supervised finetuning scheme allows us to adapt to the k-space data at test time, which, however, decreases the quantitative metrics, although generating the visually most textured and sharp images. For this challenge, we focused on robust and high SSIM scores, which we achieved by ensembling all models to a $\Sigma$-net.
BibTeX
@misc{schlemper2019net,
  author = {Jo Schlemper and Chen Qin and Jinming Duan and Ronald M. Summers and Kerstin Hammernik},
  title = {{Σ-net: Ensembled Iterative Deep Neural Networks for Accelerated Parallel MR Image Reconstruction}},
  year = {2019},
  eprint = {1912.05480},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1912.05480},
}
Figure