For decades, the standard approach to "inverse problems"—recovering a clean signal from noisy, blurry, or incomplete measurements—relied on the assumption of sparsity. If we knew a signal was sparse in some basis (like wavelets for images), we could reconstruct it from surprisingly few measurements. However, sparsity is a relatively weak prior. It captures the fact that images have many zeros in certain domains, but it doesn't capture what makes a face look like a face.
A new paradigm has emerged that replaces hand-crafted sparsity with deep generative priors. Instead of assuming a signal is sparse, we assume it lies within the range of a pre-trained generative model like a GAN, VAE, or Diffusion model. This shift allows for much higher fidelity reconstructions, but it introduces a significant challenge: the optimization landscape becomes a non-convex minefield.
The Core Mechanism: Generative Inversion
The fundamental idea, popularized by researchers like Manik Dhar, Aditya Grover, and Stefano Ermon in Modeling Sparse Deviations for Compressed Sensing using Generative Models, is to treat the generative model $G$ as a constraint.
If we have a measurement $y = Ax + \eta$ (where $A$ is a sensing matrix and $\eta$ is noise), we assume the true signal $x$ can be represented as $G(z)$ for some latent code $z$. The reconstruction task then becomes an optimization problem: find the $z$ that minimizes the difference between the observed measurements and the measurements we would get from the generated image:
$$\min_{z} \| y - A G(z) \|^2$$
As Pei Peng and Shirin Jalali demonstrate in Using auto-encoders for solving ill-posed linear inverse problems, this approach can theoretically recover a signal with a number of measurements proportional only to the latent dimension $k$, regardless of the ambient signal dimension.
The Problem: The Non-Convex Minefield
While generative priors are powerful, they are mathematically "unruly." Traditional sparsity-based optimization is usually convex, meaning any local minimum is a global minimum. In contrast, the mapping $G(z)$ in a deep network involves non-linearities (like ReLU) and multiple layers that create a highly non-convex loss surface.
As noted by Subarna Tripathi in Precise Recovery of Latent Vectors from Generative Adversarial Networks, standard gradient descent often gets "stuck" in local optima. This leads to several failure modes:
- Identity Loss: In facial inpainting, the model might generate a realistic face that doesn't actually belong to the person in the original photo Jain et al..
- Visual Artifacts: The optimizer may settle on a latent code that produces "hallucinations" or distorted features Zheng et al..
- Posterior Collapse: In VAEs, the optimizer might find a trivial solution where the latent code is ignored entirely Menon et al..
Strategies for Navigating the Landscape
Researchers have developed several distinct strategies to bypass these local minima and reach the global optimum.
1. Intermediate Layer Optimization (ILO)
Instead of only optimizing the initial latent code $z$, Giannis Daras and Joseph Dean proposed Intermediate Layer Optimization. By progressively optimizing the activations of deeper layers in the generator, the algorithm gains more degrees of freedom to fit the measurements while staying close to the "natural image manifold." This prevents the optimizer from being bottlenecked by the limited expressivity of the initial latent space.
2. Amortized Optimization (AIPO)
Tianci Liu, Qi Lei, and colleagues introduced Amortized Inverse Problem Optimization (AIPO) in Optimization for Amortized Inverse Problems. Rather than solving the hard problem all at once, AIPO solves a sequence of tasks. It starts with a simple Maximum Likelihood Estimate (where the prior is weak) and gradually increases the influence of the generative prior. This "curriculum" guides the optimizer toward the global minimum, effectively smoothing the non-convex landscape.
3. Joint Posterior Maximization (JPMAP)
Traditional methods optimize only $z$. However, Andres Almansa and Mario González argue in Solving Inverse Problems by Joint Posterior Maximization with a VAE Prior that we should optimize both the image $x$ and the latent code $z$ simultaneously. This "quasi-bi-convex" formulation is more stable and less likely to fall into the spurious local optima that plague pure latent-space searches.
4. Advanced Sampling and Message Passing
For high-stakes or high-dimensional settings, gradient descent may be insufficient. Parthe Pandit and Sundeep Rangan developed ML-VAMP Inference With Deep Generative Priors in High Dimensions, an algorithm that uses message passing to achieve Bayes-optimal estimation. Similarly, Alex Schwing and Tiantian Fang utilize Hamiltonian Monte Carlo to navigate "ragged" energy landscapes where gradients are too sharp for standard methods Co-Generation with GANs using AIS based HMC.
Beyond Standard Reconstruction
This framework is now being extended to more complex statistical tasks. For example, Zhaoqiang Liu and Jonathan Scarlett introduced Generative PCA Generative Principal Component Analysis, which replaces the sparsity assumption in PCA with a generative prior, allowing for the recovery of principal components from far fewer samples than previously thought possible.