QuantArt: Quantizing Image Style Transfer Towards High Visual Fidelity

Siyu Huang, Jie An, Donglai Wei, Jiebo Luo, Hanspeter Pfister

Introduction

Arbitrary Style Transfer (AST) has traditionally been viewed as a balancing act between two competing objectives: preserving the content of a source photograph and adopting the stylistic elements of a reference artwork. However, even when models achieve high statistical similarity to a reference style, the resulting images often suffer from "uncanny" textures or digital artifacts that betray their AI origins. QuantArt addresses this by introducing visual fidelity as a third critical dimension, ensuring that stylized outputs not only match a specific style but also align with the broader distribution of real-world art.

The Visual Fidelity Gap

Existing AST algorithms typically minimize a hybrid loss function that balances content and style. The authors argue that this binary focus ignores the inherent "artistic-ness" of the output. Without a mechanism to ground the generation in the space of real art, models often produce results that look like imitations rather than genuine paintings. QuantArt introduces vector quantization to anchor generated features to a learned "vocabulary" of real artwork, producing images that are significantly harder for humans to distinguish from genuine paintings.

Conceptual framework comparing traditional style transfer to QuantArt, alongside the visual impact of quantization.

The QuantArt Framework

The core innovation of QuantArt is the application of Vector Quantization (VQ) to the style transfer pipeline. The process involves three key components:

  1. Artistic Codebooks: The model uses VQ-VAE-style auto-encoders to learn discrete codebooks for both photos ($Z_{photo}$) and artworks ($Z_{art}$). These codebooks act as a global dictionary of representative feature patches found in real images.
  2. Style-Guided Attention (SGA): This module employs self-attention to capture global content structures and cross-attention to map these structures to the style reference.
  3. Quantized Refinement: During inference, stylized features are "pushed" toward the nearest centroids in the learned artwork codebook. This quantization step acts as a powerful regularizer, snapping unrealistic or noisy features to patterns that exist in actual training data.

The framework provides two control knobs: $\beta$ (Style Fidelity), which dictates resemblance to the specific reference, and $\alpha$ (Visual Fidelity), which controls the degree of quantization. A higher $\alpha$ forces the image to look more like "real art" generally, even if it deviates slightly from the specific reference style.

Experimental Results

QuantArt was evaluated across several domains using standard benchmarks, including general artistic transfer (MS-COCO/WikiArt), face stylization (FFHQ/MetFaces), and landscape transfer (LandscapesHQ).

Qualitative results across various domains, demonstrating the versatility of the QuantArt framework.

Quantitative Performance

QuantArt achieved state-of-the-art results in Fréchet Inception Distance (FID) and ArtFID, metrics that assess how well the generated distribution matches the target artistic distribution. In Photo-to-Art tasks, QuantArt reached an FID of 17.787, significantly outperforming competitors like AdaAttN (25.894) and StyTR2 (30.893).

Human Perception and the "Confusion Test"

To measure visual fidelity directly, the authors conducted a "Turing Test" for art. Participants were shown pairs of images—one real painting and one QuantArt output—and asked to identify the genuine artwork. On average, participants were only correct 51.6% of the time, suggesting that QuantArt's outputs are nearly indistinguishable from real paintings to the average viewer. Furthermore, in head-to-head user preference studies, QuantArt was consistently favored over other leading methods by a wide margin.

Conclusion and Future Work

By introducing quantization into the style transfer pipeline, QuantArt effectively bridges the gap between AI generation and artistic realism. While high levels of quantization can occasionally lead to a loss of fine-grained detail from the reference style, the gain in overall visual fidelity is substantial. Future work may explore hierarchical quantization and varying codebook sizes to further refine the balance between specific style adherence and general artistic quality.

Summary updated

Abstract

The mechanism of existing style transfer algorithms is by minimizing a hybrid loss function to push the generated image toward high similarities in both content and style. However, this type of approach cannot guarantee visual fidelity, i.e., the generated artworks should be indistinguishable from real ones. In this paper, we devise a new style transfer framework called QunntArt for high visual-fidelity stylization. QuantArt pushes the latent representation of the generated artwork toward the centroids of the real artwork distribution with vector quantization. By fusing the quantized and continuous latent representations, QuantArt allows flexible control over the generated artworks in terms of content preservation, style similarity, and visual fidelity. Experiments on various style transfer settings show that our QuantArt framework achieves significantly higher visual fidelity compared with the existing style transfer methods.
BibTeX
@inproceedings{huang2022quantart,
  author = {Siyu Huang and Jie An and Donglai Wei and Jiebo Luo and Hanspeter Pfister},
  title = {{QuantArt: Quantizing Image Style Transfer Towards High Visual Fidelity}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
}
Figure