Abstract
Recent advancements in image restoration have leveraged discrete generative priors, or "codebooks," to enhance robustness against noise and degradation. However, existing methods often struggle with the trade-off between category-specific specialization and universal applicability. This paper introduces AdaCode, a novel framework that replaces static, fixed codebooks with image-adaptive discrete priors. By learning to blend multiple specialized basis codebooks through a predicted spatial weight map, AdaCode achieves state-of-the-art performance in super-resolution and inpainting across diverse natural scenes without requiring separate models for different image categories.
Introduction
Discrete generative priors, popularized by models like VQGAN, have become a cornerstone of high-fidelity image reconstruction. By mapping continuous image features to a finite set of learned "visual words," these models provide a powerful inductive bias that helps reconstruct clean textures from degraded inputs.
Despite their success, current codebook-based methods face a significant limitation: the diversity of the natural world. A codebook optimized for human faces lacks the specific vocabulary required to reconstruct architectural details or natural landscapes. Conversely, "universal" codebooks that attempt to cover all categories often suffer from limited expressiveness, leading to blurry textures or unnatural artifacts. AdaCode addresses this by introducing a class-agnostic restoration framework that dynamically adapts its prior to the specific content of the input image.
The AdaCode Framework
The core innovation of AdaCode is the transition from a single static codebook to a weighted combination of multiple basis codebooks. The training process is divided into three distinct stages to ensure both specialization and adaptability.
Stage I: Basis Codebook Pretraining
To capture the breadth of visual textures found in nature, the researchers first divided a high-quality dataset into five semantic "super-classes": Architectures, Indoor objects, Natural scenes, Street views, and Portraits. A separate VQGAN-style basis codebook was trained for each category. This specialization ensures that each basis contains a highly efficient vocabulary for its specific domain.
Stage II: AdaCode Representation Learning
With the basis codebooks fixed, the model introduces a Weight Predictor based on Swin Transformer blocks. For any given input image, this module predicts a spatial weight map. The final latent representation is not a single entry from one codebook, but a weighted linear combination of entries from all five basis codebooks. This "soft" blending allows the model to interpolate between discrete codes, significantly increasing the expressiveness of the prior and allowing it to represent complex, hybrid textures that do not fit neatly into a single category.
Stage III: Restoration via AdaCode
In the final stage, the adaptive codebook and decoder are utilized for downstream tasks such as Super-Resolution (SR) and Inpainting. The model employs a code-level loss—incorporating InfoNCE and style loss—to align degraded image features with their high-resolution counterparts within the adaptive latent space. This ensures that the restoration process remains faithful to the learned high-quality priors.
Experimental Results
Quantitative Performance
AdaCode was evaluated against several state-of-the-art (SOTA) benchmarks across reconstruction, super-resolution, and inpainting tasks:
- Image Reconstruction: On the texture-rich OST dataset, AdaCode achieved a PSNR of 25.76 dB, significantly outperforming the standard VQGAN (21.36 dB).
- Super-Resolution: In $4\times$ upscaling tests on the Urban100 benchmark, AdaCode reached a PSNR of 21.45 dB, surpassing Real-ESRGAN (21.35 dB) and FeMaSR (20.51 dB).
- Inpainting: On the DIV2K validation set, AdaCode achieved an LPIPS score of 0.0516, demonstrating superior perceptual similarity compared to MAT (0.0676) and GPEN (0.0933).
Qualitative Insights
Beyond numerical gains, AdaCode demonstrates a unique ability to handle multi-content scenes. For example, in an image featuring a person in front of a building, the weight map can locally assign the "Portrait" basis to the face and the "Architecture" basis to the background. Furthermore, the weighted combination mechanism alleviates the "discontinuity" problem inherent in purely discrete quantization, preserving fine structures that traditional codebooks often miss.
Discussion and Future Work
AdaCode represents a significant step toward truly class-agnostic image restoration. By leveraging non-multicollinear bases, the model captures diverse perspectives of the latent space and adapts them to the input on the fly.
However, several avenues for future research remain. The current selection of five basis codebooks is empirical; determining the optimal number and type of bases remains an open question. Additionally, while the three-stage training process is effective, moving toward a fully end-to-end approach could further optimize performance. Finally, integrating high-level semantic segmentation could refine the weight prediction process, moving beyond low-level feature analysis to more context-aware adaptation.
Conclusion
By moving from static to adaptive discrete priors, AdaCode provides a flexible and expressive framework for image restoration. It effectively bridges the gap between specialized category-specific models and general-purpose restorers, setting a new standard for high-fidelity, class-agnostic image processing.
Summary updated