Introduction
Arbitrary Style Transfer (AST) has traditionally been viewed as a balancing act between two competing objectives: preserving the content of a source photograph and adopting the stylistic elements of a reference artwork. However, even when models achieve high statistical similarity to a reference style, the resulting images often suffer from "uncanny" textures or digital artifacts that betray their AI origins. QuantArt addresses this by introducing visual fidelity as a third critical dimension, ensuring that stylized outputs not only match a specific style but also align with the broader distribution of real-world art.
The Visual Fidelity Gap
Existing AST algorithms typically minimize a hybrid loss function that balances content and style. The authors argue that this binary focus ignores the inherent "artistic-ness" of the output. Without a mechanism to ground the generation in the space of real art, models often produce results that look like imitations rather than genuine paintings. QuantArt introduces vector quantization to anchor generated features to a learned "vocabulary" of real artwork, producing images that are significantly harder for humans to distinguish from genuine paintings.
The QuantArt Framework
The core innovation of QuantArt is the application of Vector Quantization (VQ) to the style transfer pipeline. The process involves three key components:
- Artistic Codebooks: The model uses VQ-VAE-style auto-encoders to learn discrete codebooks for both photos ($Z_{photo}$) and artworks ($Z_{art}$). These codebooks act as a global dictionary of representative feature patches found in real images.
- Style-Guided Attention (SGA): This module employs self-attention to capture global content structures and cross-attention to map these structures to the style reference.
- Quantized Refinement: During inference, stylized features are "pushed" toward the nearest centroids in the learned artwork codebook. This quantization step acts as a powerful regularizer, snapping unrealistic or noisy features to patterns that exist in actual training data.
The framework provides two control knobs: $\beta$ (Style Fidelity), which dictates resemblance to the specific reference, and $\alpha$ (Visual Fidelity), which controls the degree of quantization. A higher $\alpha$ forces the image to look more like "real art" generally, even if it deviates slightly from the specific reference style.
Experimental Results
QuantArt was evaluated across several domains using standard benchmarks, including general artistic transfer (MS-COCO/WikiArt), face stylization (FFHQ/MetFaces), and landscape transfer (LandscapesHQ).
Quantitative Performance
QuantArt achieved state-of-the-art results in Fréchet Inception Distance (FID) and ArtFID, metrics that assess how well the generated distribution matches the target artistic distribution. In Photo-to-Art tasks, QuantArt reached an FID of 17.787, significantly outperforming competitors like AdaAttN (25.894) and StyTR2 (30.893).
Human Perception and the "Confusion Test"
To measure visual fidelity directly, the authors conducted a "Turing Test" for art. Participants were shown pairs of images—one real painting and one QuantArt output—and asked to identify the genuine artwork. On average, participants were only correct 51.6% of the time, suggesting that QuantArt's outputs are nearly indistinguishable from real paintings to the average viewer. Furthermore, in head-to-head user preference studies, QuantArt was consistently favored over other leading methods by a wide margin.
Conclusion and Future Work
By introducing quantization into the style transfer pipeline, QuantArt effectively bridges the gap between AI generation and artistic realism. While high levels of quantization can occasionally lead to a loss of fine-grained detail from the reference style, the gain in overall visual fidelity is substantial. Future work may explore hierarchical quantization and varying codebook sizes to further refine the balance between specific style adherence and general artistic quality.
Summary updated