PILOT: Physics-Informed Learned Optimized Trajectories for Accelerated MRI

Tomer Weiss, Ortal Senouf, Sanketh Vedula, Oleg Michailovich, Michael Zibulevsky, Alex BronsteinThe Journal of Machine Learning for Biomedical Imaging 2021 · DOI

TL;DR

This paper introduces PILOT, a novel deep learning framework that jointly optimizes MRI k-space trajectories (how the scanner collects data) and image reconstruction. By treating trajectory design and reconstruction as a single, end-to-end learning problem, PILOT achieves significantly faster MRI scans with improved image quality, while respecting the physical constraints of MRI hardware.

What Problem Does This Paper Address?

Magnetic Resonance Imaging (MRI) is a powerful diagnostic tool, but its long acquisition times are a major drawback. These long scans can lead to motion artifacts, increase costs, and limit its use in time-sensitive situations like emergency radiology or dynamic imaging. While researchers have explored accelerating MRI through optimized data acquisition (k-space trajectories) or improved reconstruction algorithms, these approaches have typically been tackled separately. This paper addresses the challenge of simultaneously optimizing both aspects to achieve greater acceleration and better image quality, while ensuring the learned trajectories are physically feasible on real MRI hardware.

Methods

The core of PILOT is a deep neural network designed for end-to-end learning. This network integrates three key components:

  • Sub-sampling Layer: This layer models the data acquisition process, generating k-space measurements along a learned trajectory. It uses a non-uniform Fast Fourier Transform (NUFFT) to sample the k-space.
  • Regridding Layer: This layer transforms the irregularly sampled k-space data into a Cartesian grid image space using an adjoint NUFFT.
  • Task Model: This component performs the specific end-task, either image reconstruction (e.g., using a U-Net architecture) or image segmentation.

Crucially, the entire pipeline is designed to be differentiable with respect to the k-space trajectory coordinates. This allows the framework to optimize the trajectory itself based on the performance of the end-task and adherence to physical constraints. The optimization objective includes both a task-specific loss (e.g., L1 norm for reconstruction, cross-entropy for segmentation) and a constraint-fitting term that penalizes violations of hardware limitations like maximum gradient strength and slew rate.

Two extensions are proposed to address the non-convexity of the optimization problem:

  • PILOT-TSP: This approach uses a Traveling Salesman Problem (TSP) solver to order learned k-space points, helping to find more globally optimal trajectories.
  • Multi-scale PILOT: This method refines the trajectory learning by starting with a coarse grid of control points and gradually increasing their density, leading to more global trajectory changes.

Datasets

The paper utilizes two datasets for training and evaluation:

  • NYU fastMRI initiative database: This dataset consists of raw knee MRI volumes acquired using fully-sampled Cartesian trajectories. It was used for the image reconstruction tasks. The dataset was split into training/validation and testing sets.
  • Medical Image Segmentation Decathlon (MSD): Specifically, the brain tumors dataset from MSD was used for the segmentation task. This dataset contains 4D brain MRI scans with gold-standard annotations. For this work, only T2-weighted images containing tumors were used, and simulated k-space data was generated from these images due to the unavailability of raw k-space data for segmentation.

Key Insights and Ideas

  • Joint Optimization is Key: The most significant insight is that optimizing the k-space trajectory and the reconstruction network simultaneously leads to substantial improvements over optimizing them separately. This holistic approach allows the system to learn trajectories that are not just "good" in isolation, but specifically tailored to produce the best possible reconstruction for a given reconstruction method.
  • Physics-Informed Learning: By explicitly incorporating hardware constraints (gradient strength, slew rate) into the learning process, PILOT ensures that the generated trajectories are physically feasible on real MRI scanners, bridging the gap between theoretical optimization and practical implementation.
  • Task-Specific Trajectories: The framework demonstrates that trajectories optimized for image reconstruction differ from those optimized for segmentation. Reconstruction trajectories tend to focus more on lower frequencies for pixel-wise similarity, while segmentation trajectories prioritize higher frequencies for edge and structural information. This opens up the possibility of designing MRI protocols tailored to specific diagnostic tasks.
  • Robustness to Noise: The learned trajectories and reconstruction models show improved resilience to lower Signal-to-Noise Ratio (SNR) conditions, suggesting that PILOT can adapt to varying scan qualities.

Key Results

  • Image Reconstruction: PILOT consistently outperformed fixed, handcrafted trajectories (spiral, radial, Cartesian) across various decimation rates (DR). Improvements in PSNR ranged from 3.26 to 5.56 dB, and SSIM improved by 0.080 to 0.170 points. PILOT-TSP further enhanced these results.
  • Image Segmentation: For tumor segmentation, learned trajectories achieved improvements of 0.015–0.044 DICE points and 0.018–0.056 IOU points compared to fixed trajectories.
  • Efficiency: The paper suggests that PILOT can achieve significantly shorter acquisition times, potentially reducing the number of shots by two to four times without compromising image quality.
  • Comparison to SPARKLING: PILOT outperformed the state-of-the-art SPARKLING method for single-shot trajectories, achieving 1.15 dB higher PSNR and 0.030 higher SSIM, attributed to PILOT's data-driven optimization of sampling density.

Limitations and Future Work

  • Retrospective Validation: The current study relied on retrospective acquisitions. The authors plan to validate the method prospectively by deploying learned trajectories on a real MRI machine.
  • Segmentation Dataset: The segmentation experiments used simulated k-space data. Obtaining raw k-space data for segmentation tasks is a future goal to provide a more direct evaluation.
  • Additional Constraints: Future work includes incorporating echo time (TE) constraints to control contrast and considering more accurate physical models, such as those accounting for eddy current effects and off-resonance effects.
  • Multi-shot PILOT-TSP: Extending the PILOT-TSP approach to multi-shot trajectories is also a planned future direction.
  • Joint Task Learning: The paper explored jointly learning segmentation and reconstruction, showing that reconstruction aids segmentation. Further investigation into multi-task learning for optimal performance across tasks is suggested.

Summary updated

Abstract

Magnetic Resonance Imaging (MRI) has long been considered to be among “the gold standards” of diagnostic medical imaging. The long acquisition times, however, render MRI prone to motion artifacts, let alone their adverse contribution to the relative high costs of MRI examination. Over the last few decades, multiple studies have focused on the development of both physical and post-processing methods for accelerated acquisition of MRI scans. These two approaches, however, have so far been addressed separately. On the other hand, recent works in optical computational imaging have demonstrated growing success of concurrent learning-based design of data acquisition and image reconstruction schemes. In this work, we propose a novel approach to the learning of optimal schemes for conjoint acquisition and reconstruction of MRI scans, with the optimization carried out simultaneously with respect to the time-efficiency of data acquisition and the quality of resulting reconstructions. To be of a practical value, the schemes are encoded in the form of general k-space trajectories, whose associated magnetic gradients are constrained to obey a set of predefined hardware requirements (as defined in terms of, e.g., peak currents and maximum slew rates of magnetic gradients). With this proviso in mind, we propose a novel algorithm for the end-to-end training of a combined acquisition-reconstruction pipeline using a deep neural network with differentiable forward- and back-propagation operators. We also demonstrate the effectiveness of the proposed solution in application to both image reconstruction and image segmentation, reporting substantial improvements in terms of acceleration factors as well as the quality of these end tasks.
BibTeX
@article{weiss2021pilot,
  author = {Tomer Weiss and Ortal Senouf and Sanketh Vedula and Oleg Michailovich and Michael Zibulevsky and Alex Bronstein},
  title = {{PILOT: Physics-Informed Learned Optimized Trajectories for Accelerated MRI}},
  journal = {The Journal of Machine Learning for Biomedical Imaging},
  year = {2021},
  doi = {10.59275/j.melba.2021-1a1f},
}
Figure