DiffGAR: Model-Agnostic Restoration from Generative Artifacts Using Image-to-Image Diffusion Models

Yueqin Yin, Lianghua Huang, Yu Liu, Kaiqi Huang

Introduction

Generative models such as StyleGAN2, Autoregressive models, and Denoising Diffusion Probabilistic Models (DDPM) have achieved remarkable success in image synthesis. However, these models often introduce "generative artifacts"—unnatural textures, checkerboard patterns, or streaky distortions—that degrade visual quality and complicate downstream applications. Restoring these images is challenging due to the non-linear nature of artifacts, the lack of paired "clean/artifact" training data, and the diversity of generator architectures.

DiffGAR (Generative Artifact Restoration) is proposed as a model-agnostic framework that treats any generator as a black box, utilizing conditional diffusion models to effectively "clean" corrupted images.

Methodology

The DiffGAR framework operates in two distinct stages to ensure both efficiency and high-quality restoration:

  1. Stage 1 (Autoencoder): The input image is compressed into a low-dimensional latent space. This reduces computational overhead while preserving essential structural information.
  2. Stage 2 (Conditional Diffusion): A U-Net-based diffusion model is conditioned on the artifact-laden image. It iteratively denoises the latent representation to produce a clean version.

The model utilizes classifier-free guidance, allowing for a tunable trade-off between pixel-level reconstruction accuracy and the overall vividness of the restored image. It can operate in a "non-blind" mode using a class token to identify specific artifact types or a "blind" mode using a [MASK] token when the artifact type is unknown.

The two-stage architecture of DiffGAR, featuring an Autoencoder and a Conditional Diffusion U-Net.

Artifact Synthesis

To overcome the lack of paired training data, the researchers developed four mechanisms to simulate common generative artifacts on clean datasets like FFHQ and AFHQ-Dog:

  • Replace Token: Swapping discrete latent tokens to mimic VQGAN or GAN errors.
  • GPT Sampling: Using high "temperature" settings in autoregressive models to create sampling errors.
  • DDIM Gaussian & Scale: Perturbing latent codes in diffusion models with noise or scaling to generate characteristic "streaky" artifacts.

Training on these synthetic artifacts allows the model to generalize remarkably well to real-world errors produced by models like StyleGAN2.

Key Results

DiffGAR significantly outperforms existing restoration methods such as GFPGAN, EdiBERT, and Real-ESRGAN across both quantitative and qualitative benchmarks.

Quantitative Performance

On the FFHQ dataset, DiffGAR achieved a state-of-the-art FID of 18.34, a substantial improvement over GFPGAN (25.5) and EdiBERT (22.74). It also demonstrated superior performance in reconstruction metrics like PSNR and SSIM.

Qualitative and Human Evaluation

Qualitatively, DiffGAR is uniquely capable of removing complex streaky artifacts that traditional methods often ignore. In a user preference study, DiffGAR's results were preferred 55% of the time, compared to 27% for GFPGAN and 18% for EdiBERT.

Quantitative and qualitative evaluation of DiffGAR compared to state-of-the-art methods on FFHQ and AFHQ datasets.

Conclusion

DiffGAR provides a robust, model-agnostic solution for restoring images corrupted by generative artifacts. By leveraging the power of conditional diffusion models and a novel artifact synthesis strategy, it achieves superior image quality and generalization. While diffusion-based inference is slower than single-pass GAN methods, the significant gains in visual fidelity and artifact removal make it a powerful tool for post-processing generative outputs. Future work will focus on optimizing inference speed and further improving identity preservation.

Summary updated

Abstract

Recent generative models show impressive results in photo-realistic image generation. However, artifacts often inevitably appear in the generated results, leading to downgraded user experience and reduced performance in downstream tasks. This work aims to develop a plugin post-processing module for diverse generative models, which can faithfully restore images from diverse generative artifacts. This is challenging because: (1) Unlike traditional degradation patterns, generative artifacts are non-linear and the transformation function is highly complex. (2) There are no readily available artifact-image pairs. (3) Different from model-specific anti-artifact methods, a model-agnostic framework views the generator as a black-box machine and has no access to the architecture details. In this work, we first design a group of mechanisms to simulate generative artifacts of popular generators (i.e., GANs, autoregressive models, and diffusion models), given real images. Second, we implement the model-agnostic anti-artifact framework as an image-to-image diffusion model, due to its advantage in generation quality and capacity. Finally, we design a conditioning scheme for the diffusion model to enable both blind and non-blind image restoration. A guidance parameter is also introduced to allow for a trade-off between restoration accuracy and image quality. Extensive experiments show that our method significantly outperforms previous approaches on the proposed datasets and real-world artifact images.
BibTeX
@inproceedings{yin2022diffgar,
  author = {Yueqin Yin and Lianghua Huang and Yu Liu and Kaiqi Huang},
  title = {{DiffGAR: Model-Agnostic Restoration from Generative Artifacts Using Image-to-Image Diffusion Models}},
  booktitle = {CSAI},
  year = {2022},
  doi = {10.1145/3577530.3577539},
}
Figure