Pseudoinverse-Guided Diffusion Models for Inverse Problems

Jiaming Song, Arash Vahdat, Morteza Mardani, Jan KautzICLR 2023 · OpenReview

Introduction

In computer vision, an inverse problem involves recovering an original, pristine image from a corrupted or partial measurement. Common examples include sharpening a blurry photo, filling in missing pixels (inpainting), or removing JPEG compression artifacts.

Historically, researchers have faced a dilemma when tackling these problems with generative AI:

  1. Problem-specific models: Training a diffusion model specifically for one task (e.g., 4x super-resolution) yields beautiful results. However, these models are "brittle"—they cannot generalize to other tasks, and training them from scratch is computationally expensive.
  2. Problem-agnostic models: Using a pre-trained, general generative model and "guiding" its output to match the corrupted measurements offers great flexibility. Unfortunately, these methods have historically struggled to match the image quality of specialized models, especially when dealing with noise or non-linear corruptions.

To close this performance gap, researchers have introduced $\Pi$GDM (Pseudoinverse-guided Diffusion Models). $\Pi$GDM is a "plug-and-play" framework that allows general-purpose diffusion models to solve complex inverse problems without any task-specific training, achieving performance parity with specialized state-of-the-art models.

How $\Pi$GDM Works

The core innovation of $\Pi$GDM lies in a new method for calculating guidance. In standard diffusion models, images are generated by starting with pure noise and iteratively "denoising" it. To solve an inverse problem, a guidance term is added at each step to nudge the model toward a solution consistent with the corrupted observation.

Architecture of Pseudoinverse-guided Diffusion Models ($\Pi$GDM)

Key technical components of the $\Pi$GDM framework include:

  • Pseudoinverse Guidance ($\Pi$G): While previous methods often used the transpose of the measurement matrix for guidance, $\Pi$GDM uses the Moore-Penrose pseudoinverse. This ensures the guidance remains mathematically consistent with the measurement model, which is crucial when the corruption process (like downsampling) discards significant information.
  • Handling Non-Differentiable Operators: The framework extends pseudoinverse logic to non-linear and non-differentiable processes. For tasks like JPEG compression, it defines a "generalized pseudoinverse" (essentially the JPEG decoding algorithm) to calculate the necessary guidance.
  • Vector-Jacobian Product (VJP): To implement this efficiently, the model calculates the gradient of the denoised estimate with respect to the current noisy state via backpropagation through the diffusion network at every sampling step.
  • Adaptive Weights (AW): Instead of a fixed guidance strength, $\Pi$GDM employs a heuristic that automatically adjusts the guidance weight based on the current timestep, significantly improving stability and image sharpness.

Key Insights and Capabilities

The use of a pseudoinverse provides a "consistency" breakthrough. Because the approximation of the image distribution doesn't depend on the specific measurement model used, $\Pi$GDM generalizes far better than previous reconstruction guidance methods. Furthermore, by incorporating the noise level of the measurement directly into the guidance formula, the model avoids overfitting to noisy measurements—a common failure mode for other plug-and-play solvers.

Perhaps the most impressive capability of $\Pi$GDM is its compositionality. Because the method is entirely agnostic to the specific problem, measurement models can be chained together.

$\Pi$GDM solving composite inverse problems like combined masking, JPEG compression, and low resolution.

As shown above, $\Pi$GDM can solve a "triple threat" problem—such as an image that is simultaneously low-resolution, missing pixels, and heavily JPEG compressed—all at once, despite never having seen that specific combination during training.

Empirical Results

Evaluated on standard datasets like ImageNet (256x256) and LSUN Bedroom, $\Pi$GDM demonstrated remarkable performance against both task-specific models (Palette, SR3) and agnostic methods (DDRM, DPS):

  • Super-Resolution: On ImageNet, $\Pi$GDM achieved an FID score of 3.2, outperforming the specialized SR3 model (5.2) and matching the specialized ADM-U (3.1).
  • JPEG Restoration: $\Pi$GDM is the first problem-agnostic diffusion method to effectively handle JPEG artifact removal, achieving results nearly identical to Palette, a model trained specifically for that exact task.
  • Step Efficiency: While $\Pi$GDM requires more computation per step, it requires far fewer total steps (as few as 20–100) to reach high quality compared to competitors like DPS, which often require upwards of 1,000 steps.

Limitations and Future Work

Despite its strong performance, $\Pi$GDM has limitations. Because it requires a backpropagation pass (the VJP) at every denoising step, each iteration is roughly three times more expensive in terms of memory and time than a standard diffusion step. While it needs fewer steps overall, it remains slower than replacement-based methods like DDRM.

Currently, this research focuses on diffusion in "pixel space." Extending these pseudoinverse principles to "latent space" models (like Stable Diffusion) represents an open and highly promising area for future research, potentially unlocking even greater efficiency and scale.

Summary updated

Abstract

Diffusion models have become competitive candidates for solving various inverse problems. Models trained for specific inverse problems work well but are limited to their particular use cases, whereas methods that use problem-agnostic models are general but often perform worse empirically. To address this dilemma, we introduce Pseudoinverse-guided Diffusion Models ($\Pi$GDM), an approach that uses problem-agnostic models to close the gap in performance. $\Pi$GDM directly estimates conditional scores from the measurement model of the inverse problem without additional training. It can address inverse problems with noisy, non-linear, or even non-differentiable measurements, in contrast to many existing approaches that are limited to noiseless linear ones. We illustrate the empirical effectiveness of $\Pi$GDM on several image restoration tasks, including super-resolution, inpainting and JPEG restoration. On ImageNet, $\Pi$GDM is competitive with state-of-the-art diffusion models trained on specific tasks, and is the first to achieve this with problem-agnostic diffusion models. $\Pi$GDM can also solve a wider set of inverse problems where the measurement processes are composed of several simpler ones.
BibTeX
@inproceedings{song2023pseudoinverse,
  author = {Jiaming Song and Arash Vahdat and Morteza Mardani and Jan Kautz},
  title = {{Pseudoinverse-Guided Diffusion Models for Inverse Problems}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=9_gsMA8MRKQ},
}
Figure