Image Restoration and Editing through Generative Latent Inversion

This direction is merged into another direction: out-of-distribution reconstruction failure. Its last version stays readable and citable here.

Version 1 of 115 papers15 conceptsfrozen 2026-10-04

Traditional image processing treats tasks like super-resolution or denoising as deterministic mathematical problems: find a function that maps a blurry input back to its sharp original. However, as Vasileios Lioutas notes in Mapping Low-Resolution Images To Multiple High-Resolution Images, these problems are fundamentally "ill-posed." A single blurry pixel could represent a dozen different fine-grained textures.

To solve this, researchers have shifted toward using generative priors. Instead of guessing pixels from scratch, we leverage pre-trained models like StyleGAN or Diffusion models that already "know" what the world looks like. The challenge lies in inversion: finding the exact point in the model's internal "latent space" that corresponds to a specific real-world image.

The Latent Space as a Semantic Map

Generative models don't just store images; they organize them into a structured manifold. In this space, distances often correspond to semantic changes rather than pixel-wise differences. Yotam Nitzan and colleagues demonstrate this in LARGE, showing that the distance of an image's latent code to a semantic boundary can predict attributes like age or pose with incredible accuracy, even with only two labeled examples.

Because the latent space is so well-organized, we can use it to "fill in the blanks" for corrupted data. For instance, Givi Meishvili et al. show in Learning to Have an Ear for Face Super-Resolution that when pixels are too blurry to identify a person, the generative prior can combine those pixels with audio cues to recover identity and gender—details that are physically absent from the low-resolution image but present in the "knowledge" of the model.

The Inversion Trade-off: Fidelity vs. Editability

To use these priors, we must map a real image into the latent space. This is typically done via an encoder, like the pixel2style2pixel (pSp) framework by Elad Richardson and Or Patashnik, which translates images directly into StyleGAN codes.

However, a central tension exists:

  1. Fidelity: Reconstructing the image exactly as it looks, including unique quirks.
  2. Editability: Keeping the code within the "natural" distribution of the model so we can still manipulate it.

If you optimize too hard for fidelity, the latent code might drift into "uncharted" areas of the latent space where the model's semantic rules break down. Jiapeng Zhu and Bolei Zhou address this in In-Domain GAN Inversion by using a domain-guided encoder. This ensures the inverted code stays within the "semantic domain," allowing for realistic edits (like changing an expression) without destroying the image's identity.

Hallucination and the Out-of-Distribution Problem

The primary risk of using a generative prior is "hallucination"—the model might replace a unique real-world feature with a generic one it saw during training. Majed El Helou and Sabine Süsstrunk tackle this in BIGPrior by explicitly decoupling hallucinated information from the actual data fidelity, providing a map of where the model is "guessing."

This problem is most acute with Out-of-Distribution (OOD) images. If a model was trained on upright faces, it might struggle to invert a face that is upside down or wearing a rare accessory. Rakshith Subramanyam et al. proposed SPHInX to mitigate this, using "vicinal regularization" to help pre-trained generators reconstruct images they were never originally intended to handle, such as cartoons or medical scans.

Beyond GANs: Diffusion and 3D Priors

While much of this work began with GANs, the field is rapidly moving toward diffusion models and 3D representations. Hamadi Chihaoui and Paolo Favaro introduced the Diffusion Image Prior (DIIP), which exploits the fact that diffusion models naturally "prefer" clean images over noisy ones. By optimizing noise to reconstruct a corrupted image, the model acts as a zero-shot restorer.

Similarly, in the realm of 3D reconstruction, Xinhang Liu and colleagues use diffusion models in Deceptive-NeRF to generate "pseudo-observations." When you only have a few photos of a scene, the generative prior "imagines" the missing viewpoints, allowing for high-quality 3D reconstructions from sparse data.

Ultimately, this research direction moves us away from simple pixel manipulation and toward a world where AI uses its collective "memory" of visual concepts to repair, enhance, and understand the images we capture.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_image_restoration_and_editing_through_ge_7376fec7,
  title = {Image Restoration and Editing through Generative Latent Inversion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/image-restoration-and-editing-through-generative-latent-inversion/rdr_7376fec7a9c54ab197784f82ac1c52f1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_image_restoration_and_editing_through_ge_7376fec7_v1,
  title = {Image Restoration and Editing through Generative Latent Inversion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/image-restoration-and-editing-through-generative-latent-inversion/rdr_7376fec7a9c54ab197784f82ac1c52f1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}