Abstract
Transformers have achieved great success in pluralistic image inpainting recently. However, existing transformer-based solutions often suffer from significant information loss due to aggressive image downsampling and pixel quantization. To address these issues, we propose PUT (Patch-based Un-quantized Transformer). PUT utilizes a patch-based auto-encoder (P-VQVAE) to avoid downsampling and an Un-Quantized Transformer (UQ-Transformer) that processes continuous feature vectors to preserve input richness. Extensive experiments demonstrate that PUT significantly outperforms state-of-the-art methods in image fidelity and diversity, particularly for large masked regions.
Introduction
Image inpainting involves filling missing regions with semantically plausible content. While Transformers excel at capturing global structures, their computational cost typically forces a trade-off: images are downsampled to low resolutions (e.g., 32x32) and pixels are quantized into a small discrete vocabulary. This leads to blurry results and lost fine-grained details that post-processing CNNs struggle to recover.
The PUT Framework
The PUT framework rethinks the transformer pipeline to minimize information loss through two core innovations.
1. P-VQVAE: Preserving Resolution
Instead of standard downsampling, the P-VQVAE partitions the image into non-overlapping patches. This prevents "information leakage" where empty masked pixels blur into valid surrounding pixels during encoding. It employs a Dual-Codebook strategy to distinguish between masked and unmasked patches and a Multi-Scale Guided Decoder (MSG-Dec) that references original unmasked features to ensure high-fidelity reconstruction.
2. UQ-Transformer: Avoiding Input Quantization
Traditional vision transformers require discrete tokens as input. The UQ-Transformer breaks this convention by accepting continuous feature vectors directly from the encoder. By treating quantization only as a prediction target rather than an input constraint, the model retains the full contextual richness of the original image.
Experimental Results
We evaluated PUT on three large-scale benchmarks: FFHQ (faces), Places2 (scenes), and ImageNet (objects), using irregular masks to simulate realistic inpainting scenarios.
Quantitative Performance
PUT achieves superior performance across standard metrics. For instance, on ImageNet with high mask ratios (40-60%), PUT reached an FID of 43.2, significantly better than the previous best transformer method, ICT (54.3).
Qualitative Superiority and Human Preference
As shown in the visual comparisons, PUT generates sharper textures and maintains better global consistency (e.g., facial symmetry) than competing methods. In human preference studies, PUT was selected as the most realistic in over 60% of cases across all datasets.
Conclusion
By eliminating the bottlenecks of downsampling and input quantization, the PUT framework sets a new standard for pluralistic image inpainting. While autoregressive generation remains computationally intensive, the gains in fidelity and diversity represent a significant step forward for high-resolution image synthesis.
Code is available at: https://github.com/liuqk3/PUT
Summary updated