Traditional deep learning approaches to image restoration often treat the process as a "black box," mapping degraded pixels to clean ones without regard for how the degradation occurred. While these models can produce visually pleasing results, they frequently suffer from a lack of physical grounding. As Filippos Kokkinos et al. note, ignoring principles like optical diffraction and point spread functions (PSFs) can lead to models that fail to generalize across different hardware configurations.
To address this, a growing body of research focuses on integrating physical laws—such as light transport, atmospheric scattering, and signal acquisition geometry—directly into neural architectures. This shift ensures that restorations are not just visually plausible, but physically consistent.
The Problem with Physics-Agnostic Modeling
Purely data-driven models often struggle when the training data does not perfectly match real-world conditions. For instance, Yurui Zhu et al. argue that physics-agnostic models fail to account for the relationship between scene depth and atmospheric degradation, leading to inconsistent results in adverse weather. Similarly, Jin Xie et al. observe that without grounding in atmospheric scattering models, dehazing systems frequently produce unnatural color distortions and artifacts.
In medical and scientific imaging, the stakes are higher. Yoseob Han et al. point out that de-aliasing MRI images solely in the image domain often results in "waxy" textures and the loss of high-frequency diagnostic details. Rui Li further highlights that standard generative models lack the constraints to ensure cellular reconstructions remain consistent with the laws of light propagation, potentially creating "hallucinations" or fake structures.
Unrolling Classical Physics into Neural Networks
One powerful way to bridge this gap is by "unrolling" traditional optimization algorithms into deep learning frameworks. Instead of replacing classical physics, these models use neural networks to learn specific parameters or regularizers within a physically sound pipeline.
- Iterative Optimization: Kristina Monakhova et al. unrolled a traditional optimization algorithm into a network for lensless imaging, achieving a 20x speedup while maintaining the physical rigor of light transport. Similarly, Hui Liu et al. unrolled gradient descent to reconstruct hyperspectral images, using a learnable "incremental gradient" to maintain accuracy with far fewer parameters.
- Hybrid Filtering: Filippos Kokkinos et al. embedded learnable regularizers directly into the classical Wiener-Kolmogorov filter, combining content-adaptive neural networks with the physical principles of microscopy.
- Geometric Reconstruction: In CT imaging, Xi Tan et al. replaced the rigid components of the Filtered Back Projection (FBP) algorithm with learnable modules, ensuring the reconstruction respects the underlying geometry of the data acquisition process.
Constraining the Generative Process
Researchers are also using physical models as "hints" or constraints to guide generative models like GANs and Diffusion models. By explicitly modeling the degradation process, these systems search a smaller, more accurate solution space.
For example, Xijun Wang et al. introduced a variational approach to "learn" the specific nature of atmospheric turbulence before attempting restoration. In underwater imaging, Pan Mu et al. used dedicated modules to estimate light transmission and color casts, preventing the over-saturation common in black-box models. Dongwei Ren et al. utilized an isotropic reblurring module to ensure that deblurring models remain physically consistent even when trained on misaligned data.
Leveraging Acquisition Physics
Finally, some researchers argue that restoration should happen where the data is born. Liutao Yang et al. developed a transformer that operates on raw CT sinograms, leveraging the physical symmetries of the projection data that are lost once the image is reconstructed. Songnan Lin et al. grounded their video deblurring network in the physical model of event sensors, using log-intensity changes to recover lost structural details.
By incorporating these domain-specific priors—whether it is the angle of polarization in Shuangfan Zhou et al. or the transient signal of light bounces in Adriano Simonetto—researchers are moving toward a paradigm where deep learning acts as a sophisticated solver for well-defined physical equations, rather than a generic statistical estimator.