Unsupervised Adversarial Image Reconstruction

Arthur Pajot, Emmanuel de Bézenac, Patrick GallinariICLR 2019 · OpenReview

In fields ranging from satellite imagery to medical scanning, we frequently encounter "dirty" data—images obscured by clouds, sensor noise, or missing pixels. Traditionally, training a neural network to fix these images requires a dataset of pairs: the corrupted image and its perfect, clean counterpart.

But what if the "clean" version doesn't exist? In many real-world scenarios, we only have access to corrupted observations. This paper tackles the unsupervised reconstruction problem: how to learn the underlying structure of a signal (like a face or a room) when you only have access to lossy, inaccurate measurements and some basic knowledge of how those measurements were corrupted.

By cleverly combining adversarial learning with the known statistics of the corruption process, the authors present a framework for reconstructing clean images from corrupted ones without ever needing to see a "clean" image during training. The resulting reconstruction quality rivals supervised methods that have access to ground-truth data.

A Two-Pronged Approach to Unsupervised Reconstruction

The authors frame the problem as a Maximum A Posteriori (MAP) estimation. Essentially, they want to find the most probable clean image given a corrupted observation. To achieve this without ground truth, they use a two-pronged approach based on Generative Adversarial Networks (GANs):

  1. Handling the Prior (The "Realistic" Look): They adopt an idea from AmbientGAN. Instead of a discriminator looking at clean images, the discriminator looks at corrupted versions of the generator's output and compares them to the corrupted real-world observations. If the generator can produce a clean image that, when corrupted, looks indistinguishable from the real noisy data, it must be learning the true underlying distribution.
  2. Handling the Likelihood (The "Consistency" Check): To ensure the reconstruction actually matches the specific input observation (and isn't just a random pretty picture), they introduce a "re-corruption" loss. They take the reconstructed image, apply a fresh instance of the known corruption process, and minimize the Mean Squared Error (MSE) between this "re-corrupted" version and the original observation.
The proposed architecture combines an adversarial discriminator to ensure realistic outputs with a consistency loss (MSE) to ensure the reconstruction matches the specific input observation.

By combining these two mechanisms, the model learns to produce images that are both statistically realistic and faithful to the original input.

Datasets and Corruptions

The model was evaluated on three standard image datasets (CelebA, LSUN Bedrooms, and Recipe-1M), all resized to 64x64. The authors applied four types of synthetic corruptions to test the model's robustness:

  • Remove-Pixel: Deleting a large fraction (up to 95%) of pixels.
  • Remove-Pixel-Channel: Deleting specific color channels at random coordinates.
  • Convolve-Noise: Applying a blur filter followed by Gaussian noise.
  • Patch-Band: Removing a horizontal strip (20 pixels high) from the image.

Key Insights and Results

The most powerful insight from this work is the reliance on process statistics. You don't need clean data if you understand the nature of your "noise." If you know the statistics of the corruption (e.g., "I'm missing 90% of pixels at random"), that is enough signal to recover the whole. Because the model learns the distribution of the data, it doesn't just "smooth" over gaps; it performs semantic inference, effectively "hallucinating" missing features in a contextually correct way.

This approach successfully closes the supervision gap. The unsupervised approach performs nearly as well as "paired" models trained with full access to ground-truth clean images, suggesting that the alignment provided by the corruption process is a sufficient substitute for explicit labels.

Quantitative comparison (MSE) of the proposed model against baselines and supervised variants across different corruption types.

As shown in the results above, the model demonstrates significant quantitative superiority. On the CelebA dataset with 95% pixel removal, the proposed model achieved a Mean Squared Error (MSE) of 0.0414, compared to 0.292 for the previous state-of-the-art unsupervised baseline (Conditional AmbientGAN). Furthermore, the model's performance was remarkably close to the "Paired Variant" (MSE 0.0383), which had the "unfair" advantage of seeing clean images during training.

Qualitatively, the model successfully removes heavy noise and fills in large missing bands while maintaining sharp edges and realistic textures, whereas simpler denoising or inpainting methods often produce blurry or semantically incorrect results.

Limitations and Future Work

While highly effective, the approach does have limitations:

  • Known Corruption Model: The primary limitation is that the model requires prior knowledge of the measurement process. If the nature of the corruption is unknown or modeled incorrectly, the reconstruction will fail.
  • Deterministic Output: The current model provides a single "best guess" reconstruction. Future versions could be made stochastic to provide a range of possible reconstructions, allowing for uncertainty estimation.

Looking ahead, the next logical step is applying this framework to real-world scientific data—such as medical or astronomical imagery—where obtaining ground truth is physically impossible, but the physics of the sensor noise are well understood.

Summary updated

Abstract

We address the problem of recovering an underlying signal from lossy, inaccurate observations in an unsupervised setting. Typically, we consider situations where there is little to no background knowledge on the structure of the underlying signal, no access to signal-measurement pairs, nor even unpaired signal-measurement data. The only available information is provided by the observations and the measurement process statistics. We cast the problem as finding the \textit{maximum a posteriori} estimate of the signal given each measurement, and propose a general framework for the reconstruction problem. We use a formulation of generative adversarial networks, where the generator takes as input a corrupted observation in order to produce realistic reconstructions, and add a penalty term tying the reconstruction to the associated observation. We evaluate our reconstructions on several image datasets with different types of corruptions. The proposed approach yields better results than alternative baselines, and comparable performance with model variants trained with additional supervision.
BibTeX
@inproceedings{pajot2019unsupervised,
  author = {Arthur Pajot and Emmanuel de Bézenac and Patrick Gallinari},
  title = {{Unsupervised Adversarial Image Reconstruction}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=lnW03pDlRio},
}
Figure