Abstract
Most contemporary image style transfer models are deterministic, producing a single static output for any given content-style pair. This paper introduces DIST (Diverse Image Style Transfer), a framework that utilizes an invertible cross-space mapping to ensure that random noise inputs translate into meaningful, high-quality artistic variations. By enforcing a bijective relationship between the latent noise space and the style space, DIST achieves significant diversity without sacrificing the structural integrity or visual quality of the original image.
Introduction
Style transfer is an inherently subjective task; there is no single "correct" way to interpret a photograph through the lens of a specific artist like Van Gogh or Monet. However, most existing Deep Convolutional Neural Network (DCNN) approaches treat style transfer as a deterministic optimization problem.
While researchers have attempted to introduce diversity by concatenating random noise vectors to content features, these models often suffer from "mode collapse" where the network ignores the noise in favor of the high-dimensional content information. Previous attempts to force diversity through explicit penalty losses often resulted in visual artifacts or structural distortions. The DIST framework addresses this challenge by ensuring that the model is both highly diverse and consistently high-quality through a novel architectural design.
Methodology
The authors propose the DIST framework, which is structured around three specialized branches designed to disentangle and manipulate style and content effectively.
1. Stylization Branch
This core generator utilizes an encoder-decoder architecture. It employs Adaptive Instance Normalization (AdaIN) to fuse content features from the photo with style features derived from the target artwork. This allows the model to apply global style statistics to the local structures of the content image.
2. Disentanglement Branch
To achieve true style transfer, the model must distinguish between "content" (the objects and layout) and "style" (the brushstrokes and color palette). This branch uses two independent encoders to extract shared content information from both the original photo and the stylized result, while a separate style encoder ($E_s$) learns to capture the artistic essence.
3. Inverse Branch: The Invertible Mapping
The key innovation of DIST is the enforcement of an invertible mapping between the input noise vector ($z$) and the style space of the generated image. By utilizing an "inverse loss" ($L_{inv}$), the model ensures that the style of the output image can be mapped back to the exact noise vector that generated it. This mathematical bijection prevents the network from ignoring the noise, guaranteeing that every unique $z$ produces a unique artistic variation.
Furthermore, the authors replace rigid pixel-wise content losses with a Content Structure Loss ($L_p$). By applying average pooling before calculating the distance between images, the model allows for "artistic deformation"—permitting brushstrokes to deviate slightly from the photo's original lines while maintaining the overall composition.
Experimental Results
The DIST framework was evaluated against seven state-of-the-art baselines, including Gatys et al., MUNIT, and AST, using the Places365 dataset for content and Wikiart for style examples.
Qualitative Comparison
As shown in the comparative analysis, DIST maintains superior structural consistency compared to methods like MUNIT, which often produce color shifts without meaningful texture changes. DIST successfully captures the intricate brushwork of the target style while allowing for diverse outputs based on different noise seeds.
Quantitative Metrics
- Deception Rate: A VGG-16 network trained to recognize artists was used to evaluate the "authenticity" of the generated art. DIST achieved a deception rate of 0.525, significantly outperforming the next best competitor (AST at 0.454).
- User Study: In a human preference test involving 50 participants, DIST was top-rated, receiving 32.1% of the votes for best style resemblance.
- Diversity (LPIPS): Using the Learned Perceptual Image Patch Similarity metric, DIST demonstrated a high balance of diversity and quality. While some models showed higher raw diversity scores, they often did so at the cost of significant image degradation, a trade-off DIST avoids.
Key Insights
- Invertibility as a Guarantee: By making the relationship between noise and style a bijection, the authors mathematically prevent the network from collapsing into a single output.
- Structural Flexibility: The realization that pixel-perfect content matching hinders artistic quality led to the development of the Content Structure Loss, allowing for more natural "paint flow."
- Latent Space Navigation: Unlike earlier methods that treated style as simple pixel statistics, DIST treats style as a navigable latent space.
Conclusion and Future Work
DIST represents a significant step forward in making style transfer a more creative and diverse tool. While the current model is specialized for image-to-image tasks, the authors suggest that the invertible mapping concept could be extended to text-to-image synthesis or image inpainting. Future work will focus on streamlining the multi-branch training process to improve inference efficiency for real-time applications.
Summary updated