Reduce Information Loss in Transformers for Pluralistic Image Inpainting

Qiankun Liu, Zhentao Tan, Dongdong Chen, Qi Chu, Xiyang Dai, Yinpeng Chen, Mengchen Liu, Lu Yuan, Nenghai Yu

Abstract

Transformers have achieved great success in pluralistic image inpainting recently. However, existing transformer-based solutions often suffer from significant information loss due to aggressive image downsampling and pixel quantization. To address these issues, we propose PUT (Patch-based Un-quantized Transformer). PUT utilizes a patch-based auto-encoder (P-VQVAE) to avoid downsampling and an Un-Quantized Transformer (UQ-Transformer) that processes continuous feature vectors to preserve input richness. Extensive experiments demonstrate that PUT significantly outperforms state-of-the-art methods in image fidelity and diversity, particularly for large masked regions.

Introduction

Image inpainting involves filling missing regions with semantically plausible content. While Transformers excel at capturing global structures, their computational cost typically forces a trade-off: images are downsampled to low resolutions (e.g., 32x32) and pixels are quantized into a small discrete vocabulary. This leads to blurry results and lost fine-grained details that post-processing CNNs struggle to recover.

The PUT Framework

The PUT framework rethinks the transformer pipeline to minimize information loss through two core innovations.

Overview of the PUT framework, illustrating the P-VQVAE and the UQ-Transformer components.

1. P-VQVAE: Preserving Resolution

Instead of standard downsampling, the P-VQVAE partitions the image into non-overlapping patches. This prevents "information leakage" where empty masked pixels blur into valid surrounding pixels during encoding. It employs a Dual-Codebook strategy to distinguish between masked and unmasked patches and a Multi-Scale Guided Decoder (MSG-Dec) that references original unmasked features to ensure high-fidelity reconstruction.

2. UQ-Transformer: Avoiding Input Quantization

Traditional vision transformers require discrete tokens as input. The UQ-Transformer breaks this convention by accepting continuous feature vectors directly from the encoder. By treating quantization only as a prediction target rather than an input constraint, the model retains the full contextual richness of the original image.

Experimental Results

We evaluated PUT on three large-scale benchmarks: FFHQ (faces), Places2 (scenes), and ImageNet (objects), using irregular masks to simulate realistic inpainting scenarios.

Qualitative comparison of PUT against leading methods (ICT, PIC, etc.) across FFHQ, Places2, and ImageNet datasets.

Quantitative Performance

PUT achieves superior performance across standard metrics. For instance, on ImageNet with high mask ratios (40-60%), PUT reached an FID of 43.2, significantly better than the previous best transformer method, ICT (54.3).

Qualitative Superiority and Human Preference

As shown in the visual comparisons, PUT generates sharper textures and maintains better global consistency (e.g., facial symmetry) than competing methods. In human preference studies, PUT was selected as the most realistic in over 60% of cases across all datasets.

Conclusion

By eliminating the bottlenecks of downsampling and input quantization, the PUT framework sets a new standard for pluralistic image inpainting. While autoregressive generation remains computationally intensive, the gains in fidelity and diversity represent a significant step forward for high-resolution image synthesis.

Code is available at: https://github.com/liuqk3/PUT

Summary updated

Abstract

Transformers have achieved great success in pluralistic image inpainting recently. However, we find existing transformer based solutions regard each pixel as a token, thus suffer from information loss issue from two aspects: 1) They downsample the input image into much lower resolutions for efficiency consideration, incurring information loss and extra misalignment for the boundaries of masked regions. 2) They quantize 256 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> RGB pixels to a small number (such as 512) of quantized pixels. The indices of quantized pixels are used as tokens for the inputs and prediction targets of transformer. Although an extra CNN network is used to upsample and refine the low-resolution results, it is difficult to retrieve the lost information back. To keep input information as much as possible, we propose a new transformer based framework “PUT”. Specifically, to avoid input downsampling while maintaining the computation efficiency, we design a patch-based auto-encoder P-VQVAE, where the encoder converts the masked image into non-overlapped patch tokens and the decoder recovers the masked regions from the inpainted tokens while keeping the unmasked regions unchanged. To eliminate the information loss caused by quantization, an Un-Quantized Transformer (UQ-Transformer) is applied, which directly takes the features from P-VQVAE encoder as input without quantization and regards the quantized tokens only as prediction targets. Extensive experiments show that PUT greatly outperforms state-of-the-art methods on image fidelity, especially for large masked regions and complex large-scale datasets.
BibTeX
@inproceedings{liu2022reduce,
  author = {Qiankun Liu and Zhentao Tan and Dongdong Chen and Qi Chu and Xiyang Dai and Yinpeng Chen and Mengchen Liu and Lu Yuan and Nenghai Yu},
  title = {{Reduce Information Loss in Transformers for Pluralistic Image Inpainting}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  url = {https://openreview.net/forum?id=dpXu1eGfXYpM},
}
Figure