From Posterior Sampling to Meaningful Diversity in Image Restoration

Noa Cohen, Hila Manor, Yuval Bahat, Tomer Michaeli

TL;DR

When restoring degraded images, standard generative models sample randomly from a "posterior distribution," which often yields nearly identical results due to the distribution's heavy-tailed nature. This paper introduces meaningfully diverse image restoration, proposing both post-processing selection strategies and a training-free diffusion guidance mechanism that explicitly forces generated images to be semantically distinct without sacrificing quality.

What Problem Does This Paper Address?

Image restoration tasks—like inpainting a missing region or super-resolving a low-resolution image—are fundamentally ill-posed. There are infinitely many valid ways to fill in a blank patch of sky or upscale a blurry face.

To show users the range of possibilities, modern state-of-the-art methods (like diffusion models) typically draw random samples from the posterior distribution (the probability of a clean image given the degraded input). However, the authors show that this strategy has a major practical flaw: the posterior distribution is often heavily imbalanced and heavy-tailed.

If you try to inpaint a missing patch of sky, the most probable completion is just empty blue sky or simple clouds. If a user generates 5 random samples, they will likely get 5 nearly identical variations of plain sky. But a meaningfully diverse tool should instead present one plain sky, one with a bird, one with a hot air balloon, and one with an airplane. Current posterior sampling methods fail to surface these rare but entirely plausible semantic alternatives because they are buried deep in the tails of the distribution.

Vanilla posterior sampling often yields redundant images, whereas meaningfully diverse sampling outlines a broader range of possibilities.

Methods

The authors tackle this problem in two ways: first, by exploring how to sub-sample a large pool of generated images, and second, by modifying the diffusion generation process directly.

1. Post-Processing Selection Baselines

If you already have a large pool of posterior samples ($\tilde{N} = 100$), how do you pick a small, diverse subset ($N = 5$) to show the user? The authors test three feature-space approaches:

  • Cluster Representatives (K-Means): Cluster the 100 images into 5 groups using deep semantic features and pick the center of each cluster.
  • Uniformization: Estimate the local density of the samples using a $k$-Nearest Neighbor ($k$-NN) estimator, and sample inversely proportional to this density to give rare modes a higher chance.
  • Farthest Point Strategy (FPS): Start with one random sample, then iteratively pick the next sample that maximizes the minimum distance to the already selected set in feature space.

2. Online Diversity-Guided Generation

Generating 100 images just to pick 5 is computationally expensive. To solve this, the authors propose a practical, training-free guidance mechanism for diffusion models.

They run $N$ diffusion processes simultaneously (in a batch). At each reverse timestep $t$, the model estimates the final clean image for each batch element. The authors introduce a diversity guidance step that computes the gradient of a dissimilarity loss. Crucially, this loss pushes each image in the batch away from its nearest neighbor in the batch:

$$\hat{x}_{0|t}^i \leftarrow \hat{x}_{0|t}^i + \eta \frac{t}{T} \nabla d(\hat{x}_{0|t}^i, \hat{x}_{0|t}^{i,NN})$$

To prevent images from distorting once they are already distinct, the distance metric is clamped beyond a threshold $D$. This entire process requires no retraining and adds negligible computational overhead.

Datasets

The methods were evaluated on two primary datasets:

  • CelebAMask-HQ: Used for face inpainting and extreme ($16\times$) super-resolution. Features were extracted using an AnyCost attribute predictor to capture semantic traits like facial expressions, hair color, and accessories.
  • PartImageNet: Used for natural image inpainting and super-resolution, utilizing VGG-16 and LPIPS metrics for feature distance evaluation.

Key Insights and Ideas

  • The Posterior is Heavy-Tailed: The authors mathematically prove the "redundancy problem" by measuring the kurtosis of the sample distributions. Real image posteriors exhibit incredibly high kurtosis compared to standard Gaussians, meaning distinct semantic alternatives behave like statistical outliers.
Projections of posterior samples show high kurtosis, indicating that alternative semantic meanings (like adding glasses or closing a mouth) reside in heavy tails.
  • FPS Outperforms Uniformization: While uniform sampling sounds ideal in theory, it tends to completely drop the dominant mode if its effective support area is small. FPS strikes the best balance: it naturally picks a sample from the high-probability dominant mode first, and then immediately jumps to the sparse, interesting regions of the feature space.

Key Results

  • Human Evaluation: Extensive user studies on Amazon Mechanical Turk showed that humans overwhelmingly prefer the diversity produced by the Farthest Point Strategy (FPS) and the online guidance method over vanilla posterior sampling.
  • Preserving Coverage: Crucially, the user studies confirmed that maximizing diversity did not come at the expense of "coverage"—the diverse sets still contained an image similar to the ground truth roughly as often as vanilla sampling did.
User study results showing that FPS and diversity guidance are perceived as significantly more diverse while maintaining high coverage of likely solutions.
  • Quantitative Improvements: Across four prominent diffusion backbones (RePaint, DDRM, DDNM, and DPS), the proposed diversity guidance consistently improved the LPIPS diversity score while maintaining comparable image quality (measured via NIQE) and data consistency (measured via LR-PSNR).

Limitations and Future Work

The current approach focuses on general meaningful diversity—it pushes images apart along any available semantic axis (e.g., changing hair color, adding glasses, or altering expressions simultaneously).

The authors note that future work should explore attribute-specific diversity. For instance, a forensic investigator might want to see 5 variations of different hats while keeping the suspect's face identical, or a radiologist might want to see different pathological interpretations of a tumor while keeping the surrounding anatomy completely invariant.

Summary updated

Abstract

Image restoration problems are typically ill-posed in the sense that each degraded image can be restored in infinitely many valid ways. To accommodate this, many works generate a diverse set of outputs by attempting to randomly sample from the posterior distribution of natural images given the degraded input. Here we argue that this strategy is commonly of limited practical value because of the heavy tail of the posterior distribution. Consider for example inpainting a missing region of the sky in an image. Since there is a high probability that the missing region contains no object but clouds, any set of samples from the posterior would be entirely dominated by (practically identical) completions of sky. However, arguably, presenting users with only one clear sky completion, along with several alternative solutions such as airships, birds, and balloons, would better outline the set of possibilities. In this paper, we initiate the study of meaningfully diverse image restoration. We explore several post-processing approaches that can be combined with any diverse image restoration method to yield semantically meaningful diversity. Moreover, we propose a practical approach for allowing diffusion based image restoration methods to generate meaningfully diverse outputs, while incurring only negligent computational overhead. We conduct extensive user studies to analyze the proposed techniques, and find the strategy of reducing similarity between outputs to be significantly favorable over posterior sampling. Code and examples are available at https://noa-cohen.github.io/MeaningfulDiversityInIR.
BibTeX
@inproceedings{cohen2024from,
  author = {Noa Cohen and Hila Manor and Yuval Bahat and Tomer Michaeli},
  title = {{From Posterior Sampling to Meaningful Diversity in Image Restoration}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=ff2g30cZxj},
}
Figure