Improving Detail in Pluralistic Image Inpainting with Feature Dequantization

Kyungri Park, Woohwan JungWACV 2025

TL;DR

This paper introduces a novel "Feature Dequantization Module" (FDM) to address the loss of detail in pluralistic image inpainting caused by feature quantization in VQGAN-based models. FDM effectively restores lost information, significantly improving the quality and consistency of inpainted images with minimal computational overhead, and can be seamlessly integrated into existing VQGAN architectures.

What Problem Does This Paper Address?

Pluralistic Image Inpainting (PII) aims to generate multiple plausible completions for missing image regions, enhancing user satisfaction. While recent VQGAN-based methods like PUT have improved structural integrity, they suffer from a critical issue: feature quantization. This process, which involves mapping continuous features to discrete codebook entries, inevitably leads to information loss. This loss degrades the detail quality of the generated images, resulting in noticeable mismatches in texture and color at the boundaries of the inpainted areas, making the final image appear unnatural. The paper tackles this problem by proposing a method to recover this lost detail information.

Methods

The core contribution is the Feature Dequantization Module (FDM). FDM works by estimating the "quantization error" – the difference between the ideal continuous feature and its quantized representation. It then adds this estimated error back to the quantized features to reconstruct a more complete representation.

The FDM is integrated into the existing VQGAN-based inpainting pipeline (specifically, the PUT model) as an additional step after feature sampling and before decoding. The FDM itself is a relatively small network composed of residual blocks.

A significant methodological innovation is the efficient training strategy for FDM. Instead of relying on the computationally expensive iterative patch-wise feature sampling during training, the authors propose training FDM using ground truth data. This approach bypasses the need for the feature sampler and decoder during FDM training, drastically reducing training time by an estimated 184 times.

Datasets

The paper uses the following datasets:

  • Places [35]: A large-scale dataset for scene recognition.
  • Paris Street View Dataset [4]: A dataset of street scenes.

For image generation tasks beyond inpainting, they also utilize:

  • FFHQ [14]: For unconditional image generation.
  • ADE20K [36]: For semantic-conditional image generation.
  • ImageNet [2]: For class-conditional image synthesis.

Key Insights and Ideas

  • Feature quantization is the bottleneck for detail quality: The paper clearly identifies feature quantization as the primary culprit behind degraded detail in VQGAN-based inpainting, not just the discrete nature of the latent space itself.
  • De-quantization as a direct solution: Instead of trying to avoid quantization or using larger codebooks (which have their own drawbacks), the paper proposes a direct approach: learning to reverse the quantization process.
  • Minimal overhead for significant gains: FDM is designed to be a lightweight module that can be added with negligible impact on inference time, making it practical for real-world applications.
  • Efficient training is crucial: The proposed ground-truth training method for FDM is a key enabler, making the module practical to train without prohibitive computational costs.
  • Generalizability beyond inpainting: The success of FDM in improving other VQGAN-based generation tasks suggests its broader applicability in addressing information loss in vector-quantized models.

Key Results

  • Improved Detail and Consistency: FDM significantly enhances the detail quality and structural consistency of inpainted images, reducing noticeable boundaries and color mismatches.
  • Quantitative Improvements: The proposed method (PUT+FDM) achieves state-of-the-art FID scores on both Places and Paris Street View datasets, particularly outperforming PUT and ICT. For example, on the Paris Street View dataset with large masks, PUT+FDM achieves an FID of 18.66, compared to PUT's 20.52.
  • Diversity Preservation: The method maintains the pluralistic nature of the inpainting results, generating diverse and natural-looking images.
  • Drastic Training Speedup: The proposed training method for FDM reduces training time by approximately 184 times compared to naive sampling-based training.
  • Minimal Inference Overhead: FDM adds only about 0.02% to the overall inference time.
  • Effective in other VQGAN tasks: Applying FDM to unconditional, semantic-conditional, and class-conditional image generation tasks also leads to improved FID scores.

Limitations and Future Work

  • Inpainting Order: The paper acknowledges that FDM is applied after feature sampling, meaning it cannot influence the order in which patches are inpainted. This order can affect the final output quality and variety.
  • Future Work Suggestion: The authors suggest configuring FDM to perform dequantization on unmasked parts of the input, allowing it to be applied during the sampling process and thus influence the inpainting order. This could potentially lead to further improvements.

Summary updated

BibTeX
@inproceedings{park2025improving,
  author = {Kyungri Park and Woohwan Jung},
  title = {{Improving Detail in Pluralistic Image Inpainting with Feature Dequantization}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure