TL;DR
When restoring degraded images, standard generative models sample randomly from a "posterior distribution," which often yields nearly identical results due to the distribution's heavy-tailed nature. This paper introduces meaningfully diverse image restoration, proposing both post-processing selection strategies and a training-free diffusion guidance mechanism that explicitly forces generated images to be semantically distinct without sacrificing quality.
What Problem Does This Paper Address?
Image restoration tasks—like inpainting a missing region or super-resolving a low-resolution image—are fundamentally ill-posed. There are infinitely many valid ways to fill in a blank patch of sky or upscale a blurry face.
To show users the range of possibilities, modern state-of-the-art methods (like diffusion models) typically draw random samples from the posterior distribution (the probability of a clean image given the degraded input). However, the authors show that this strategy has a major practical flaw: the posterior distribution is often heavily imbalanced and heavy-tailed.
If you try to inpaint a missing patch of sky, the most probable completion is just empty blue sky or simple clouds. If a user generates 5 random samples, they will likely get 5 nearly identical variations of plain sky. But a meaningfully diverse tool should instead present one plain sky, one with a bird, one with a hot air balloon, and one with an airplane. Current posterior sampling methods fail to surface these rare but entirely plausible semantic alternatives because they are buried deep in the tails of the distribution.
Methods
The authors tackle this problem in two ways: first, by exploring how to sub-sample a large pool of generated images, and second, by modifying the diffusion generation process directly.
1. Post-Processing Selection Baselines
If you already have a large pool of posterior samples ($\tilde{N} = 100$), how do you pick a small, diverse subset ($N = 5$) to show the user? The authors test three feature-space approaches:
- Cluster Representatives (K-Means): Cluster the 100 images into 5 groups using deep semantic features and pick the center of each cluster.
- Uniformization: Estimate the local density of the samples using a $k$-Nearest Neighbor ($k$-NN) estimator, and sample inversely proportional to this density to give rare modes a higher chance.
- Farthest Point Strategy (FPS): Start with one random sample, then iteratively pick the next sample that maximizes the minimum distance to the already selected set in feature space.
2. Online Diversity-Guided Generation
Generating 100 images just to pick 5 is computationally expensive. To solve this, the authors propose a practical, training-free guidance mechanism for diffusion models.
They run $N$ diffusion processes simultaneously (in a batch). At each reverse timestep $t$, the model estimates the final clean image for each batch element. The authors introduce a diversity guidance step that computes the gradient of a dissimilarity loss. Crucially, this loss pushes each image in the batch away from its nearest neighbor in the batch:
$$\hat{x}_{0|t}^i \leftarrow \hat{x}_{0|t}^i + \eta \frac{t}{T} \nabla d(\hat{x}_{0|t}^i, \hat{x}_{0|t}^{i,NN})$$
To prevent images from distorting once they are already distinct, the distance metric is clamped beyond a threshold $D$. This entire process requires no retraining and adds negligible computational overhead.
Datasets
The methods were evaluated on two primary datasets:
- CelebAMask-HQ: Used for face inpainting and extreme ($16\times$) super-resolution. Features were extracted using an AnyCost attribute predictor to capture semantic traits like facial expressions, hair color, and accessories.
- PartImageNet: Used for natural image inpainting and super-resolution, utilizing VGG-16 and LPIPS metrics for feature distance evaluation.
Key Insights and Ideas
- The Posterior is Heavy-Tailed: The authors mathematically prove the "redundancy problem" by measuring the kurtosis of the sample distributions. Real image posteriors exhibit incredibly high kurtosis compared to standard Gaussians, meaning distinct semantic alternatives behave like statistical outliers.
- FPS Outperforms Uniformization: While uniform sampling sounds ideal in theory, it tends to completely drop the dominant mode if its effective support area is small. FPS strikes the best balance: it naturally picks a sample from the high-probability dominant mode first, and then immediately jumps to the sparse, interesting regions of the feature space.
Key Results
- Human Evaluation: Extensive user studies on Amazon Mechanical Turk showed that humans overwhelmingly prefer the diversity produced by the Farthest Point Strategy (FPS) and the online guidance method over vanilla posterior sampling.
- Preserving Coverage: Crucially, the user studies confirmed that maximizing diversity did not come at the expense of "coverage"—the diverse sets still contained an image similar to the ground truth roughly as often as vanilla sampling did.
- Quantitative Improvements: Across four prominent diffusion backbones (RePaint, DDRM, DDNM, and DPS), the proposed diversity guidance consistently improved the LPIPS diversity score while maintaining comparable image quality (measured via NIQE) and data consistency (measured via LR-PSNR).
Limitations and Future Work
The current approach focuses on general meaningful diversity—it pushes images apart along any available semantic axis (e.g., changing hair color, adding glasses, or altering expressions simultaneously).
The authors note that future work should explore attribute-specific diversity. For instance, a forensic investigator might want to see 5 variations of different hats while keeping the suspect's face identical, or a radiologist might want to see different pathological interpretations of a tumor while keeping the surrounding anatomy completely invariant.
Summary updated