X-GANs: Image Reconstruction Made Easy for Extreme Cases

Longfei Liu, Sheng Li, Yisong Chen, Guoping Wang

Abstract

Image reconstruction, including restoration and denoising, remains a significant challenge in image computing. We present X-GANs, a novel method for reconstructing images from arbitrarily corrupted sources using a variant of conditional generative adversarial networks (cGANs). Our approach utilizes a specialized generator and multi-scale discriminators, supported by a combined loss function that integrates VGG perceptual loss, adversarial perceptual loss, and a unique corresponding point loss. X-GANs excel in extreme cases—such as 95% color noise or 1% sparse sampling—where traditional methods fail. Extensive experiments on CelebA and SUN397 datasets demonstrate that X-GANs produce significantly more realistic results than existing state-of-the-art models.

Introduction

Image reconstruction is a fundamental task in computer vision, yet most existing models struggle when faced with "extreme" conditions. The authors identify three primary challenges:

  1. Extreme Sparsity: Performance drops sharply when more than 75% of data is missing.
  2. Complex Color Noise: Real-world random RGB noise is significantly harder to remove than standard Gaussian noise.
  3. Scale and Stability: Maintaining global structural coherence in high-resolution images (256x256 and above) is difficult when working from sparse local data.

X-GANs aim to "imagine" missing content by learning underlying data distributions, enabling high-fidelity reconstruction even from minimal information.

Methodology

The X-GAN framework is built upon a conditional GAN architecture with several structural and mathematical innovations.

Architecture

The Generator uses a residual architecture with downsampling modules, nine residual blocks for feature representation, and deconvolutional layers for upsampling. To handle higher resolutions, Multi-scale Discriminators evaluate the image at three different scales simultaneously. This allows the network to maintain a large receptive field, ensuring both global harmony and local textural detail.

Architecture of X-GANs featuring the generator, multi-scale discriminators, and integrated loss functions.

Loss Functions

The "X" in X-GANs refers to the multi-dimensional loss strategy:

  • Adversarial Loss: The core minimax objective.
  • VGG Perceptual Loss: Ensures semantic similarity to the ground truth.
  • Adversarial Matching Loss: Matches intermediate feature statistics within the discriminators.
  • Corresponding Point Loss: A modified L2 loss that masks the output, penalizing the model only for its reconstruction of known pixels. This prevents the model from being unfairly punished for its "hallucinations" in unknown regions.

Results and Insights

X-GANs demonstrate remarkable resilience in scenarios previously considered unrecoverable.

Extreme Performance

  • Denoising: The model can restore the primary content of an image even when 95% of pixels are replaced by random color noise.
  • Sparsity: Recognizable faces can be generated from just 1% of original pixels. At 10% sampling, results are nearly indistinguishable from the ground truth.
  • Inpainting: In white-block inpainting tasks, X-GANs achieved a PSNR of 24.99dB, significantly outperforming previous state-of-the-art baselines.
Reconstruction results across various extreme corruption scenarios for natural scenes and human faces.

Key Insights

The authors found that Edge-Aware Sampling (using a Sobel operator to prioritize high-gradient areas) significantly improves reconstruction quality compared to random sampling. Furthermore, testing with "cluttered blocks" (filling missing areas with random image fragments) proved that X-GANs can effectively distinguish between useful signal and interference.

Conclusion

X-GANs push the boundaries of image reconstruction by treating it as a generative "imagination" task rather than a simple filtering problem. While training is computationally intensive, the potential applications are vast, ranging from forensic image recovery to high-ratio image compression. Future work may integrate saliency maps to further optimize which data points are preserved during extreme compression.

Summary updated

Abstract

Image reconstruction including image restoration and denoising is a challenging problem in the field of image computing. We present a new method, called X-GANs, for reconstruction of arbitrary corrupted resource based on a variant of conditional generative adversarial networks (conditional GANs). In our method, a novel generator and multi-scale discriminators are proposed, as well as the combined adversarial losses, which integrate a VGG perceptual loss, an adversarial perceptual loss, and an elaborate corresponding point loss together based on the analysis of image feature. Our conditional GANs have enabled a variety of applications in image reconstruction, including image denoising, image restoration from quite a sparse sampling, image inpainting, image recovery from the severely polluted block or even color-noise dominated images, which are extreme cases and haven't been addressed in the status quo. We have significantly improved the accuracy and quality of image reconstruction. Extensive perceptual experiments on datasets ranging from human faces to natural scenes demonstrate that images reconstructed by the presented approach are considerably more realistic than alternative work. Our method can also be extended to handle high-ratio image compression.
BibTeX
@misc{liu2018x,
  author = {Longfei Liu and Sheng Li and Yisong Chen and Guoping Wang},
  title = {{X-GANs: Image Reconstruction Made Easy for Extreme Cases}},
  year = {2018},
  eprint = {1808.04432},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1808.04432},
}
Figure