Introduction
Generative models such as StyleGAN2, Autoregressive models, and Denoising Diffusion Probabilistic Models (DDPM) have achieved remarkable success in image synthesis. However, these models often introduce "generative artifacts"—unnatural textures, checkerboard patterns, or streaky distortions—that degrade visual quality and complicate downstream applications. Restoring these images is challenging due to the non-linear nature of artifacts, the lack of paired "clean/artifact" training data, and the diversity of generator architectures.
DiffGAR (Generative Artifact Restoration) is proposed as a model-agnostic framework that treats any generator as a black box, utilizing conditional diffusion models to effectively "clean" corrupted images.
Methodology
The DiffGAR framework operates in two distinct stages to ensure both efficiency and high-quality restoration:
- Stage 1 (Autoencoder): The input image is compressed into a low-dimensional latent space. This reduces computational overhead while preserving essential structural information.
- Stage 2 (Conditional Diffusion): A U-Net-based diffusion model is conditioned on the artifact-laden image. It iteratively denoises the latent representation to produce a clean version.
The model utilizes classifier-free guidance, allowing for a tunable trade-off between pixel-level reconstruction accuracy and the overall vividness of the restored image. It can operate in a "non-blind" mode using a class token to identify specific artifact types or a "blind" mode using a [MASK] token when the artifact type is unknown.
Artifact Synthesis
To overcome the lack of paired training data, the researchers developed four mechanisms to simulate common generative artifacts on clean datasets like FFHQ and AFHQ-Dog:
- Replace Token: Swapping discrete latent tokens to mimic VQGAN or GAN errors.
- GPT Sampling: Using high "temperature" settings in autoregressive models to create sampling errors.
- DDIM Gaussian & Scale: Perturbing latent codes in diffusion models with noise or scaling to generate characteristic "streaky" artifacts.
Training on these synthetic artifacts allows the model to generalize remarkably well to real-world errors produced by models like StyleGAN2.
Key Results
DiffGAR significantly outperforms existing restoration methods such as GFPGAN, EdiBERT, and Real-ESRGAN across both quantitative and qualitative benchmarks.
Quantitative Performance
On the FFHQ dataset, DiffGAR achieved a state-of-the-art FID of 18.34, a substantial improvement over GFPGAN (25.5) and EdiBERT (22.74). It also demonstrated superior performance in reconstruction metrics like PSNR and SSIM.
Qualitative and Human Evaluation
Qualitatively, DiffGAR is uniquely capable of removing complex streaky artifacts that traditional methods often ignore. In a user preference study, DiffGAR's results were preferred 55% of the time, compared to 27% for GFPGAN and 18% for EdiBERT.
Conclusion
DiffGAR provides a robust, model-agnostic solution for restoring images corrupted by generative artifacts. By leveraging the power of conditional diffusion models and a novel artifact synthesis strategy, it achieves superior image quality and generalization. While diffusion-based inference is slower than single-pass GAN methods, the significant gains in visual fidelity and artifact removal make it a powerful tool for post-processing generative outputs. Future work will focus on optimizing inference speed and further improving identity preservation.
Summary updated