In traditional image restoration, the goal is typically to find the "best" clean image given a corrupted observation. Whether the task is denoising, deblurring, or super-resolution, standard algorithms usually aim for a single point estimate—a deterministic mapping from the noisy input to a reconstructed output.
However, image restoration is fundamentally "ill-posed." As Kadkhodaie et al. and Bendel et al. point out, many different high-quality images could plausibly result in the same blurry or noisy observation. When we force a model to pick just one answer using traditional metrics like Mean Squared Error (MSE), the model hedges its bets by averaging all possible valid solutions. This "regression to the mean" results in the characteristic blurriness seen in many AI-restored images.
A growing body of research suggests a different path: instead of finding one average solution, we should sample from the posterior distribution of all possible solutions.
The Problem with Averages
To understand why deterministic models produce blurry results, consider the "Rate-Distortion-Perception Tradeoff" described by the authors. They demonstrate a mathematical conflict between distortion (how close pixels are to the ground truth) and perception (how natural the image looks).
When a model is trained to minimize MSE, it converges on the statistical mean of the posterior distribution. As Wan et al. explain, this is a one-to-many problem; because the model cannot decide which specific high-frequency detail is correct, it outputs a smooth average of all of them. To achieve high perceptual quality, Ohayon et al. argue that we must use stochastic processes to select a single, sharp realization from the distribution of potential reconstructions rather than an average.
Generative Models as Stochastic Solvers
To sample from these complex distributions, researchers have turned to generative architectures that can map a single corrupted input to a range of plausible outputs.
Variational Autoencoders (VAEs)
One approach uses Conditional VAEs to learn a latent space of clean images. Prakash et al. introduced DivNoising, which predicts a distribution of possible clean images. By sampling from this latent space, the model generates diverse versions of the data, which is particularly useful when signal information is partially destroyed. Similarly, Du et al. and Fu et al. use variational frameworks to handle the inherent ambiguity in tasks like rain removal and underwater enhancement, where human-defined "ground truths" are often subjective.
Diffusion and Score-Based Models
More recently, diffusion models have been adapted for posterior sampling. Chung et al. developed a method to "hijack" the reverse diffusion process, starting from a noisy image and using a low-frequency constraint to ensure the samples remain consistent with the original data. Wang et al. took this further by introducing a "variance-scaled" process. By adjusting a single mathematical "knob" ($\lambda$) at inference time, users can smoothly transition from a conservative, blurry MSE estimate to a sharp, highly realistic reconstruction without retraining the model.
Navigating Uncertainty
The primary advantage of stochastic restoration is the ability to quantify uncertainty. In high-stakes fields like medical imaging, a single "hallucinated" detail could lead to a misdiagnosis.
Piening et al. demonstrate that generative models can flag areas where the AI is unsure, providing a nuanced representation of pathological features. By generating multiple plausible reconstructions, systems like those proposed by Zhuang and Dobigeon et al. provide "credibility intervals"—a measure of how much we should trust specific pixels in the restored image.
This shift from "finding the answer" to "exploring the possibilities" allows models to handle visual ambiguity more gracefully. As Lin et al. show in the context of deraining, modeling pixel status as a continuous probability distribution allows a system to explore multiple restoration hypotheses, effectively separating noise from fine object textures that might otherwise be blurred away.
The Intuition of Sampling
Why does adding noise (stochasticity) lead to a cleaner image? It seems counterintuitive, but as Rissanen et al. explain, reversing a degradation process (like blurring or heat dissipation) is inherently unstable. By injecting controlled noise during the reconstruction, we allow the model to "jump" between different high-probability states.
Instead of getting stuck in a middle-ground average that satisfies no one, the model picks a specific, sharp path. As Ko et al. and Guo et al. suggest, averaging these diverse, sharp samples often yields a final result that is more robust and higher quality than any single deterministic guess could ever be.
Key Foundational Work
Predicting distributions with Linearizing Belief Networks Yann Dauphin (2016) — contributed 1 concept elements to this direction.
Large Scale Variational Bayesian Inference for Structured Scale Mixture Models (2012) — contributed 1 concept elements to this direction.
Variational Autoencoder for Deep Learning of Images, Labels and Captions (2016) — contributed 1 concept elements to this direction.