Abstract
Inverse problems in imaging, such as super-resolution and inpainting, are inherently ill-posed, meaning a single degraded observation can correspond to an infinite number of valid high-resolution reconstructions. Traditional deep learning methods typically aim to find a single "best" solution, often failing to capture the full range of possibilities. We present e-GLASS (exploring GAN LAtent Space Solutions), a method that navigates the latent geometry of Generative Adversarial Networks (GANs) to identify multiple valid solutions. By computing the local sensitivity of the latent space relative to both measurements and perceptual features, e-GLASS identifies "measurement-neutral" directions that allow for rapid exploration of the solution space. Our results demonstrate that e-GLASS is an order of magnitude faster than previous optimization-based methods while maintaining strict consistency with the original measurements.
Introduction
In computational imaging, an inverse problem involves recovering an original image $x$ from a degraded observation $y$. These problems are frequently ill-posed; for instance, a low-resolution image could have been generated from many different high-resolution sources. While modern generative models can produce visually stunning reconstructions, they often provide a single point estimate that reflects the model's internal biases rather than the diversity of the underlying solution space.
Existing techniques to find multiple solutions, such as PULSE, rely on repeated random restarts of complex optimization processes. This is computationally prohibitive for real-time or large-scale applications. e-GLASS addresses this by treating the GAN latent space as a geometric manifold, allowing researchers to "walk" through the space of valid solutions systematically and efficiently.
Methodology: Navigating the Latent Manifold
The e-GLASS framework operates by exploiting the local geometry of a pre-trained GAN's latent space ($z$). The process involves four primary steps:
- Initial Inversion: The process begins with a single valid reconstruction found via standard GAN inversion techniques.
- Hessian Analysis: The method computes two Hessian matrices to describe the local geometry around the initial solution:
- Measurement Sensitivity ($H_Y$): Identifies directions in the latent space that change the resulting measurements (e.g., altering the low-resolution pixels).
- Perceptual Sensitivity ($H_X$): Uses the Learned Perceptual Image Patch Similarity (LPIPS) metric to identify directions that change the semantic appearance of the image to a human observer.
- Null Space Projection: The core innovation lies in finding directions that have a high impact on $H_X$ (maximizing visual diversity) but are projected onto the null space of $H_Y$ (minimizing change to the measurements). These vectors represent changes to the image that the "sensor" cannot see.
- Geometric Exploration: Rather than re-solving the optimization problem, e-GLASS simply "nudges" the latent vector $z$ along these discovered directions, generating new, valid solutions almost instantaneously.
Experimental Results
The researchers evaluated e-GLASS across two distinct domains using specialized GAN architectures:
- Image Super-Resolution: Using BigGAN trained on ImageNet, the model downsampled 256x256 images to 32x32 and explored the possible high-resolution counterparts.
- Face Inpainting: Using PGGAN trained on CelebA-HQ, the model reconstructed missing sections of 1024x1024 facial images.
Performance and Efficiency
e-GLASS achieved a significant computational advantage, performing roughly 10 times faster than the PULSE algorithm. In a benchmark generating 10 solutions for a super-resolution task, e-GLASS required approximately 3,700 seconds, whereas PULSE required 35,000 seconds. Despite this speed, e-GLASS maintained high fidelity, keeping the $\ell_2$ measurement error below a strict feasibility threshold of $10^{-2}$.
Qualitative Diversity
The method successfully decoupled fidelity from semantics. In super-resolution tasks, e-GLASS could generate different breeds of dogs or varying fur textures that all compressed down to the exact same low-resolution input. In face inpainting, the model generated variations in facial hair, lip shapes, and skin textures within the masked area while leaving the unmasked portions of the face entirely unchanged.
Discussion and Limitations
While e-GLASS provides a powerful tool for exploring uncertainty in inverse problems, it is subject to certain constraints. The quality of the exploration is highly dependent on the initial GAN inversion; if the starting point is in a poor local minimum, the resulting diversity may be limited. Furthermore, the method is bounded by the GAN’s capacity—it can only explore features that the model has seen during training.
Currently, the framework is optimized for linear inverse problems. Future work will focus on extending these geometric principles to non-linear degradations and further refining the initial inversion process to ensure a broader starting point for exploration.
Conclusion
e-GLASS demonstrates that the latent space of GANs contains a rich, structured geometry that can be exploited to solve complex imaging problems. By turning a search problem into a geometric one, e-GLASS provides a path toward faster, more diverse, and more controllable image reconstruction.
Summary updated