Scientific and medical imaging modalities—such as Magnetic Resonance Imaging (MRI), Positron Emission Tomography (PET), and Computed Tomography (CT)—face a fundamental physical trade-off: acquiring high-fidelity data requires time. Long scan times lead to patient discomfort, motion artifacts, and high clinical costs. Traditionally, accelerating these scans meant collecting less data (undersampling) and using complex mathematical frameworks like Compressed Sensing to fill in the missing information. However, these classical approaches rely on iterative optimization algorithms that are computationally expensive, often taking minutes or hours to reconstruct a single volume.
Deep learning shifts this paradigm by moving the heavy computational burden from online optimization during the scan to offline training before deployment. By learning structural and physical priors from large datasets, neural networks can reconstruct high-fidelity images from highly undersampled data in a single feed-forward pass.
Shifting from Online Optimization to Feed-Forward Inference
To understand why deep learning revolutionizes image reconstruction, we must look at how it changes the computational workflow. As reviewed by Gregory Ongie et al. in Deep Learning Techniques for Inverse Problems in Imaging, traditional reconstruction treats every scan as an isolated inverse problem. An optimization algorithm iteratively adjusts a candidate image until its simulated scan matches the observed, undersampled data. This process must be repeated from scratch for every patient.
In contrast, a deep neural network acts as a direct feed-forward function. During training, the network learns the underlying structural priors of the target domain from thousands of high-quality reference images. When presented with highly corrupted or undersampled raw data from a new scan, the network maps the input directly to a high-fidelity reconstruction in milliseconds. This synergy, also discussed by Yuan Fang et al. in Overview of Compressed Sensing: Sensing Model, Reconstruction Algorithm, and Its Applications, allows models to bypass the slow, iterative loops of classical solvers without sacrificing spatial resolution or diagnostic utility.
Incorporating Physics and Domain Knowledge
Early deep learning models operated as pure black boxes, mapping raw inputs to outputs without considering the physical laws of image acquisition. Modern approaches achieve superior accuracy by embedding the physics of the imaging system directly into the network architecture, a technique often referred to as physics-based unrolling.
For instance, in MRI, data is collected in the frequency domain, known as k-space. Rather than working solely on the final image, advanced models process data across both domains simultaneously. In Attention Incorporated Network for Sharing Low-rank, Image and K-space Information during MR Image Reconstruction to Achieve Single Breath-hold Cardiac Cine Imaging, Siying Xu et al. introduced A-LIKNet, which uses attention mechanisms to share information between the image and k-space domains, preserving critical anatomical details like myocardial boundaries. Similarly, Zimeng Li et al. developed the EN$^2$ model in Encoding Enhanced Complex CNN for Accurate and Highly Accelerated MRI, replacing standard convolution kernels with 1D kernels designed to mimic the physical encoding of MRI scanners.
Furthermore, medical imaging data is frequently complex-valued, containing both magnitude and phase information. Dongwook Lee et al. demonstrated in Deep Residual Learning for Accelerated MRI using Magnitude and Phase Networks that training separate networks for magnitude and phase effectively eliminates aliasing artifacts. For dynamic 3D cardiac imaging, Thomas Küstner leveraged complex-valued 4D spatio-temporal convolutions in CINENet: deep learning-based 3D cardiac CINE MRI reconstruction with multi-coil complex-valued 4D spatio-temporal convolutions to complete high-resolution reconstructions in seconds, demonstrating that neural networks can replace computationally expensive algorithms in time-sensitive environments.
Collapsing Multi-Step Processing Pipelines
Beyond accelerating standard reconstructions, deep learning can collapse multi-stage analytical pipelines into a single end-to-end inference step. A prime example is Quantitative Susceptibility Mapping (QSM), a technique that measures tissue magnetic properties but traditionally requires error-prone, sequential steps like phase unwrapping and background field removal.
In Instant tissue field and magnetic susceptibility mapping from MRI raw phase using Laplacian enhanced deep neural networks, Yang Gao et al. introduced a Laplacian-enhanced layer that maps raw phase data directly to susceptibility maps in just 130 milliseconds, bypassing the traditional multi-step pipeline entirely. The authors of Quantitative susceptibility mapping using deep neural network: QSMnet similarly showed that 3D U-Nets can deliver high-quality brain maps from a single scan rather than lengthy multi-scan protocols. To push efficiency further, Jinwei Zhang et al. proposed the LARO framework in LARO: Learned Acquisition and Reconstruction Optimization to accelerate Quantitative Susceptibility Mapping, which jointly optimizes the physical k-space sampling pattern alongside a temporal-aware deep reconstruction network, achieving an 8x reduction in scan time.
Clinical Translation and Patient Comfort
The ultimate validation of these acceleration techniques lies in their clinical utility. By reducing the amount of raw data required, deep learning enables dramatic scan time reductions, directly translating to improved patient comfort and higher clinical throughput.
In cardiac imaging, where patient breath-holds limit scan duration, Pedro Ferreira et al. demonstrated in Accelerating Cardiac Diffusion Tensor Imaging With a U‐Net Based Model: Toward Single Breath‐Hold that a U-Net model can slash cardiac diffusion tensor imaging times by up to 90%. As noted by Tim Leiner et al. in Machine learning in cardiovascular magnetic resonance: basic concepts and applications, deep learning can maintain diagnostic quality even with five-fold undersampled data, turning a ten-minute procedure into a two-minute scan.
This paradigm extends to functional imaging like PET, where scan times are traditionally long due to the time required to collect sufficient photon counts. René Hosch et al. developed a Generative Adversarial Network (GAN) in Artificial intelligence guided enhancement of digital PET: scans as fast as CT? to denoise ultra-low-count data, achieving a 33-fold reduction in acquisition time. For pediatric patients, where motion and sedation risks are paramount, Qiyang Zhang et al. demonstrated a tenfold reduction in scan duration using multimodal deep learning in Reducing pediatric total-body PET/CT imaging scan time with multimodal artificial intelligence technology, allowing scans to be performed without sedation.
Generalization Across Imaging Domains
The core mechanism—leveraging deep learning to infer high-fidelity structures from sparse or noisy measurements—is highly generalizable across diverse scientific fields beyond medical imaging.
- Ultrasound: In ultrafast ultrasound imaging, Dimitris Perdios et al. utilized a residual CNN in CNN-Based Image Reconstruction Method for Ultrafast Ultrasound Imaging to match the resolution and contrast of multi-acquisition synthetic aperture methods from a single plane wave capture.
- Microscopy: In optical microscopy, Vytautas Žičkus et al. showed in Fluorescence lifetime imaging with a megapixel SPAD camera and neural network lifetime estimation that neural networks can estimate fluorescence lifetimes 1,000 times faster than traditional least-squares fitting, enabling near-real-time observation of dynamic biological processes.
- Endomicroscopy: In ultra-miniaturized endomicroscopy, David Dang et al. combined dual-comb interferometry with a specialized Transformer model in Dual-Comb Ghost Imaging with Transformer-Based Reconstruction for Optical Fiber Endomicroscopy to achieve high-fidelity reconstructions at an extreme 0.29% sampling ratio.
By replacing iterative mathematical solvers with physics-informed, feed-forward neural networks, researchers across these fields have proven that hardware limitations can be effectively mitigated by software intelligence.