When we try to reconstruct an image from indirect measurements—whether mapping X-ray attenuation in a CT scan, tracking acoustic echoes in an ultrasound, or decoding scattered photons in non-line-of-sight imaging—we are solving an inverse problem. Historically, fields relied on classical physics-based algorithms to solve these problems. More recently, end-to-end deep learning models have attempted to replace them entirely.
However, both extremes have a major flaw. Classical analytical methods are mathematically rigorous but break down when data is sparse, noisy, or highly incomplete. Conversely, purely data-driven neural networks can hallucinate realistic details, but they are "black boxes" that waste immense computational capacity learning well-known physical laws (like wave propagation or geometric projections) completely from scratch.
The emerging consensus across computational imaging is a hybrid paradigm: hybrid analytical-neural pipelines. By splitting the workload, these architectures use established physical models to handle deterministic geometric mappings, while leveraging neural networks to resolve non-linear noise, fill in missing data, and learn complex structural priors.
Domain Mapping via Two-Stage Pipelines
The most intuitive way to build a hybrid pipeline is to separate the reconstruction into two distinct sequential stages: one operating in the raw sensor (signal) domain, and the other operating in the reconstructed image domain. This can be implemented in two ways depending on where the bottleneck lies.
1. Physics First, Neural Second (Post-Processing)
In this configuration, a classical, fast analytical operator maps raw sensor data into a rough, artifact-heavy initial image. Because this step handles the complex spatial and temporal coordinate mapping, the subsequent neural network does not need to learn the laws of physics; it only needs to learn how to clean up artifacts.
For instance, in ultrafast ultrasound imaging, Dimitris Perdios et al. developed a pipeline where a traditional backprojection algorithm handles the initial geometric mapping of echoes. A deep residual network then post-processes this image, focusing exclusively on suppressing non-linear artifacts.
Raw Sensor Data ──> [ Classical Physics Solver ] ──> Rough Image ──> [ Neural Network ] ──> High-Fidelity Image
This strategy is highly effective across diverse imaging modalities:
- Elastic Source Imaging: Jaejun Yoo et al. use a classical time-reversal algorithm to generate an initial, artifact-heavy image from sparse wave data, which a Dual-Frame U-Net then refines to eliminate streaking.
- Electrical Impedance Tomography (EIT): To handle highly incomplete boundary data, Alexander Denker et al. feed multiple low-fidelity reconstructions from classical linearized solvers into a U-Net, allowing the network to focus on feature fusion and error correction rather than learning raw physics.
- Radio Frequency (RF) 3D Sensing: Jingzhi Hu et al. employ a pseudo-inverse of the physical measurement matrix to establish a grounded environmental estimate, which a neural network subsequently refines into a high-fidelity 3D reconstruction.
2. Neural First, Physics Second (Data Completion)
When sensor data is severely limited—such as when a CT scanner can only capture a few angles—the physical inversion itself becomes highly ill-posed. Here, the pipeline is inverted: a neural network operates directly in the signal domain to "fill in" the missing measurements before a classical algorithm performs the final spatial mapping.
In limited-angle CT reconstruction, Rushil Anirudh et al. use a neural network to perform implicit sinogram completion. Once the network fills in the missing views, a standard, deterministic Filtered Back Projection (FBP) algorithm computes the final image. This ensures that the final output strictly adheres to the known physics of X-ray attenuation.
Similarly, in optoacoustic imaging with limited-view data, Anna Klimovskaia Susmelj et al. use a neural network for signal imputation in the raw acoustic domain, leaving the final spatial reconstruction to classical wave propagation models.
Unrolling and Alternating Iterative Frameworks
While two-stage pipelines are computationally efficient, they lack feedback between the physical model and the neural model. A more sophisticated approach interleaves physics and neural networks iteratively, a technique often derived from algorithm unrolling.
Instead of running a classical optimization algorithm (like gradient descent) for hundreds of steps with a hand-crafted regularizer, researchers "unroll" the optimization into a fixed number of iterations and replace the regularizer with a neural network layer.
Initial Estimate ──> [ Physics Step ] ──> [ Neural Denoise ] ──> [ Physics Step ] ──> [ Neural Denoise ] ──> Final Image
▲ ▲
└─────────────────── Iterative Loop ───────┘
In mask-based lensless imaging, Kristina Monakhova et al. demonstrated that alternating between physics-based light transport iterations and deep denoising stages suppresses structured artifacts far better than purely mathematical or purely data-driven methods.
This alternating logic is also highly effective for balancing global noise reduction with structural accuracy. In low-dose CT reconstruction, the SUPER framework proposed by Zhipeng Li et al. stacks supervised deep learning layers with unsupervised iterative modules. The supervised layers focus on global noise suppression, while the unsupervised iterative modules enforce physical consistency and structural refinement.
Similarly, for phase retrieval in ptychography, Simon Welker et al. combined deep learning projections with classical physics-based algorithms. The neural network eliminates large-scale artifacts during early iterations, allowing the classical physics steps to focus on refining high-frequency details.
Embedding Neural Networks as Physical Priors
Another powerful paradigm avoids changing the classical optimization loop entirely. Instead, it uses neural networks to learn complex statistical priors or parameters, which are then plugged directly into classical frameworks like Maximum a Posteriori (MAP) or variational optimization.
Learning the Prior
In classical imaging, researchers used hand-crafted mathematical priors like Total Variation to enforce smoothness. Modern hybrid methods replace these with learned priors. For spectral compressive imaging, Tao Huang et al. integrated a Deep Gaussian Scale Mixture prior directly into a MAP framework. This allows the system to reconstruct intricate spatial edges while maintaining strict adherence to the physical laws of the imaging system. A similar strategy is utilized by Alessandra Serianni et al. in Diffuse Optical Tomography (DOT), where a data-driven autoencoder prior is paired with a physics-informed learned forward model to overcome the severe ill-posedness of tissue imaging.
Learning the Parameters
Alternatively, the neural network can be tasked with isolating a single non-linear physical property, leaving the rest of the reconstruction to a strict physical solver.
In photometric depth super-resolution, Bjoern Haefner et al. use a convolutional neural network solely to estimate surface albedo from RGB images. This estimated albedo is then treated as a known parameter inside a classical variational framework that optimizes the final 3D geometry based on the laws of optics.
RGB-D Data ──> [ CNN ] ──> Estimated Albedo ──> [ Variational Physical Framework ] ──> Precise 3D Geometry
This concept of parameterizing physical models spans several other domains:
- Optical Non-Line-of-Sight (NLOS) Imaging: Mariko Isogawa et al. use neural networks to optimize the internal parameters of a traditional Fourier-based deconvolution model, preserving light transport physics while correcting for real-world calibration errors.
- Motion Deblurring: Janne Mustaniemi integrates physical motion priors derived from smartphone gyroscope data directly into a deep network, constraining the deblurring process to physically plausible camera trajectories and eliminating ringing artifacts.
- Low-Light Restoration: The authors of INFWIDE: Image and Feature Space Wiener Deconvolution Network for Non-blind Image Deblurring in Low-Light Conditions embed classical Wiener deconvolution within a dual-branch neural network to ground low-light restoration in established optical principles.
- Dictionary Learning: Junge Zhang et al. combine deep learning with dictionary-based methods, using learned metrics to weight dictionary atoms for robust super-resolution.
Hybridizing the Algorithm Itself
Rather than wrapping a network around an algorithm, some approaches inject learnable parameters directly into the analytical equations. For example, Xi Tan et al. developed DeepFBP, which takes the classic Filtered Back Projection pipeline and replaces its rigid, hand-designed filtering and interpolation steps with learnable neural modules. The architecture retains the exact geometric trajectory of the Radon transform but allows data-driven adaptation at every stage.
Why It Works: The Power of the Null Space
To understand why hybrid pipelines consistently outperform both pure physics and pure deep learning, it helps to think in terms of linear algebra and the null space.
In an ill-posed inverse problem, multiple different images could produce the exact same sensor measurements. The difference between these possible images lies in the null space of the forward operator—the components of the image that the sensor completely fails to capture.
- Pure physics models cannot recover the null space. When data is missing, they must rely on generic mathematical assumptions (like "the image is smooth"), which leads to blurring or severe streaking artifacts.
- Pure neural networks try to learn the entire mapping, including the parts physics already explains perfectly. Because they are unconstrained, they often hallucinate details in the visible space that contradict the raw sensor data.
- Hybrid pipelines use the physical model to perfectly reconstruct everything in the visible space, ensuring the output matches the sensor data. Simultaneously, they use the neural network to intelligently fill in the null space based on learned real-world priors.
By grounding the reconstruction in physical geometry and using data-driven capacity only where the physics is blind, hybrid pipelines achieve higher fidelity, faster convergence, and robust generalization.
Key Foundational Work
Artifact Reduction Based on Sinogram Interpolation for the 3D Reconstruction of Nanoparticles Using Electron Tomography (2017) — contributed 1 concept elements to this direction.
Image super-resolution using two-channel convolutional neural networks (2016) — contributed 1 concept elements to this direction.