When restoring a degraded image—whether it is blurry, low-light, or heavily compressed—traditional algorithms face a fundamental mathematical limit. Because multiple clean images can degrade into the exact same low-quality input, the problem is highly ill-posed.
Historically, models solved this by minimizing pixel-level distances (like L1 or L2 loss). While this approach guarantees structural safety, it forces the model to average all possible clean outcomes, resulting in safe but blurry reconstructions.
The recent integration of generative priors, such as Latent Diffusion Models (LDMs) and Generative Adversarial Networks (GANs), fundamentally changed this paradigm. Instead of averaging possibilities, generative models pull the reconstruction toward a learned "natural image manifold," producing sharp, visually stunning details.
However, this creative power introduces a new pathology: generative hallucination. When an input is severely degraded, the generative prior overrides the weak underlying signal, synthesizing textures, text, or facial features that are visually plausible but factually incorrect.
Modern image restoration research focuses on bridging this gap, developing mechanisms to exploit the realism of generative priors while enforcing strict fidelity to the physical truth of the scene.
The Fidelity-Realism Trade-Off
To understand why generative models hallucinate, we must look at how they balance two competing forces: perceptual quality (how real the image looks) and distortion (how closely it matches the ground truth). As demonstrated by Zeyu Yan et al. in their work on Optimally Controllable Perceptual Lossy Compression, these two objectives are often fundamentally at odds.
When a diffusion model is frozen and used purely as a generative prior, it interprets any ambiguity or error from its guidance mechanisms as legitimate image features. As Jinshan Pan et al. point out in FaithDiff: Unleashing Diffusion Priors for Faithful Image Super-resolution, a frozen prior optimized only for synthesis will gladly manufacture sharp, incorrect details based on these misinterpretations.
[Degraded Input] ---> [Guidance Encoder] ---> [Ambiguous Signal] ---> [Frozen Generative Prior] ---> [Sharp but Hallucinated Output]
Without explicit data-consistency constraints, the model prioritizes the aesthetic conventions of its training data over the geometric reality of the input image. This results in artifacts like garbled text in document processing, altered facial identities in portrait restoration, or synthetic textures in natural scenes.
Anchoring the Diffusion Path
One of the most effective ways to mitigate hallucination is to redesign the generative process so that it is inherently bound to the degraded input, rather than relying on post-hoc guidance.
In standard diffusion models, the forward process destroys an image by turning it into pure Gaussian noise, and the reverse process learns to generate an image from scratch. In Image Restoration with Mean-Reverting Stochastic Differential Equations, Ziwei Luo et al. alter this trajectory. Instead of drifting toward random noise, the forward process treats the degraded image as a "mean" state that the system naturally reverts to.
By training a stochastic differential equation (SDE) specifically for this mean-reverting path, the reverse process is structurally anchored to the degraded input from the very first step. In a follow-up work, Photo-Realistic Image Restoration in the Wild with Controlled Vision-Language Models, Ziwei Luo et al. demonstrate that this specialized training prioritizes pixel-perfect fidelity over the unconstrained generation of general-purpose models like Stable Diffusion.
Similarly, Zalan Fabian et al. introduce DiracDiffusion: Denoising and Incremental Reconstruction with Assured Data-Consistency, which models the forward diffusion process as the actual physical degradation itself. This mathematical formulation guarantees data consistency throughout the sampling steps, allowing users to stop the process early to trade away unnecessary perceptual sharpness in exchange for absolute data accuracy.
Decoupling Imagination from Observation
Another prominent strategy is to explicitly separate the restoration network into two distinct components: one that handles structural regression (the "facts") and one that manages generative synthesis (the "imagination").
The authors of Towards Decoupling Learned Prior Hallucination and Data Fidelity in Image Restoration pioneered this with BIGPrior. Instead of letting a generative network output pixels directly, their framework learns a per-pixel fusion map. This map dynamically decides, for every coordinate in the image, whether to trust the raw data fidelity of the input or the hallucinated details of the generative prior. If the prior generates a texture that contradicts the observed data, the fusion map filters it out, significantly boosting reconstruction accuracy.
A similar philosophy underpins UGPNet, proposed by Hwayoon Lee in UGPNet: Universal Generative Prior for Image Restoration. By fusing features from a regression-based restoration network with a generative prior, the architecture preserves the identity and geometric accuracy of the subject while still benefiting from the high-frequency realism of generative features.
[Degraded Input]
│
├──> [Regression Network] ───> [Structural Base (Safe/Blurry)] ──┐
│ ├─> [Fusion Mechanism] ─> [Faithful & Sharp Output]
└──> [Generative Prior] ───> [Synthetic Details (Sharp)] ──┘
To reduce the computational burden of this dual-processing approach, models like DiffIR, developed by Bin Xia et al. in DiffIR: Efficient Diffusion Model for Image Restoration, run the diffusion process entirely within a compact latent space. By optimizing a latent prior rather than raw pixels, the model captures necessary high-frequency details without granting the network the spatial freedom required to manifest large-scale hallucinations.
Grounding Priors with Multimodal Context
When an image is so severely degraded that the remaining pixel signal is nearly zero—such as an object hidden in deep shadow or a heavily blurred sign—how can a model know which details are accurate and which are hallucinations? The emerging solution is to ground the generative prior using external, multimodal context.
In The Power of Context: How Multimodality Improves Image Super-Resolution, Kangfu Mei et al. demonstrate that integrating spatial context—such as depth maps, segmentation boundaries, and text captions—drastically curtails generative freedom. When a diffusion model is constrained by a segmentation map, it can no longer hallucinate a texture that belongs to a different semantic class, forcing the generated details to align with the true structure of the scene.
This concept of semantic grounding is also applied to real-world environments through retrieval-augmented frameworks. In ReFIR: Grounding Large Restoration Models with Retrieval Augmentation, Bin Chen et al. address the problem of weak signals by retrieving similar, high-quality reference images from an external database. By injecting the verified textures of these reference images into the restoration process, they ensure that the large restoration model remains anchored to realistic, factually grounded textures rather than relying on its own unconstrained internal imagination.