Diverse Image Style Transfer via Invertible Cross-Space Mapping

Haibo Chen, Lei Zhao, Huiming Zhang, Zhizhong Wang, Zhiwen Zuo, Ailin Li, Wei Xing, Dongming Lu

Abstract

Most contemporary image style transfer models are deterministic, producing a single static output for any given content-style pair. This paper introduces DIST (Diverse Image Style Transfer), a framework that utilizes an invertible cross-space mapping to ensure that random noise inputs translate into meaningful, high-quality artistic variations. By enforcing a bijective relationship between the latent noise space and the style space, DIST achieves significant diversity without sacrificing the structural integrity or visual quality of the original image.

Introduction

Style transfer is an inherently subjective task; there is no single "correct" way to interpret a photograph through the lens of a specific artist like Van Gogh or Monet. However, most existing Deep Convolutional Neural Network (DCNN) approaches treat style transfer as a deterministic optimization problem.

While researchers have attempted to introduce diversity by concatenating random noise vectors to content features, these models often suffer from "mode collapse" where the network ignores the noise in favor of the high-dimensional content information. Previous attempts to force diversity through explicit penalty losses often resulted in visual artifacts or structural distortions. The DIST framework addresses this challenge by ensuring that the model is both highly diverse and consistently high-quality through a novel architectural design.

Methodology

The authors propose the DIST framework, which is structured around three specialized branches designed to disentangle and manipulate style and content effectively.

1. Stylization Branch

This core generator utilizes an encoder-decoder architecture. It employs Adaptive Instance Normalization (AdaIN) to fuse content features from the photo with style features derived from the target artwork. This allows the model to apply global style statistics to the local structures of the content image.

2. Disentanglement Branch

To achieve true style transfer, the model must distinguish between "content" (the objects and layout) and "style" (the brushstrokes and color palette). This branch uses two independent encoders to extract shared content information from both the original photo and the stylized result, while a separate style encoder ($E_s$) learns to capture the artistic essence.

3. Inverse Branch: The Invertible Mapping

The key innovation of DIST is the enforcement of an invertible mapping between the input noise vector ($z$) and the style space of the generated image. By utilizing an "inverse loss" ($L_{inv}$), the model ensures that the style of the output image can be mapped back to the exact noise vector that generated it. This mathematical bijection prevents the network from ignoring the noise, guaranteeing that every unique $z$ produces a unique artistic variation.

The DIST framework architecture featuring the Stylization, Disentanglement, and Inverse branches.

Furthermore, the authors replace rigid pixel-wise content losses with a Content Structure Loss ($L_p$). By applying average pooling before calculating the distance between images, the model allows for "artistic deformation"—permitting brushstrokes to deviate slightly from the photo's original lines while maintaining the overall composition.

Experimental Results

The DIST framework was evaluated against seven state-of-the-art baselines, including Gatys et al., MUNIT, and AST, using the Places365 dataset for content and Wikiart for style examples.

Qualitative Comparison

As shown in the comparative analysis, DIST maintains superior structural consistency compared to methods like MUNIT, which often produce color shifts without meaningful texture changes. DIST successfully captures the intricate brushwork of the target style while allowing for diverse outputs based on different noise seeds.

Qualitative comparison of DIST against state-of-the-art style transfer methods.

Quantitative Metrics

  • Deception Rate: A VGG-16 network trained to recognize artists was used to evaluate the "authenticity" of the generated art. DIST achieved a deception rate of 0.525, significantly outperforming the next best competitor (AST at 0.454).
  • User Study: In a human preference test involving 50 participants, DIST was top-rated, receiving 32.1% of the votes for best style resemblance.
  • Diversity (LPIPS): Using the Learned Perceptual Image Patch Similarity metric, DIST demonstrated a high balance of diversity and quality. While some models showed higher raw diversity scores, they often did so at the cost of significant image degradation, a trade-off DIST avoids.

Key Insights

  • Invertibility as a Guarantee: By making the relationship between noise and style a bijection, the authors mathematically prevent the network from collapsing into a single output.
  • Structural Flexibility: The realization that pixel-perfect content matching hinders artistic quality led to the development of the Content Structure Loss, allowing for more natural "paint flow."
  • Latent Space Navigation: Unlike earlier methods that treated style as simple pixel statistics, DIST treats style as a navigable latent space.

Conclusion and Future Work

DIST represents a significant step forward in making style transfer a more creative and diverse tool. While the current model is specialized for image-to-image tasks, the authors suggest that the invertible mapping concept could be extended to text-to-image synthesis or image inpainting. Future work will focus on streamlining the multi-branch training process to improve inference efficiency for real-time applications.

Summary updated

Abstract

Image style transfer aims to transfer the styles of artworks onto arbitrary photographs to create novel artistic images. Although style transfer is inherently an underdetermined problem, existing approaches usually assume a deterministic solution, thus failing to capture the full distribution of possible outputs. To address this limitation, we propose a Diverse Image Style Transfer (DIST) framework which achieves significant diversity by enforcing an invertible cross-space mapping. Specifically, the framework consists of three branches: disentanglement branch, inverse branch, and stylization branch. Among them, the disentanglement branch factorizes artworks into content space and style space; the inverse branch encourages the invertible mapping between the latent space of input noise vectors and the style space of generated artistic images; the stylization branch renders the input content image with the style of an artist. Armed with these three branches, our approach is able to synthesize significantly diverse stylized images without loss of quality. We conduct extensive experiments and comparisons to evaluate our approach qualitatively and quantitatively. The experimental results demonstrate the effectiveness of our method.
BibTeX
@inproceedings{chen2021diverse,
  author = {Haibo Chen and Lei Zhao and Huiming Zhang and Zhizhong Wang and Zhiwen Zuo and Ailin Li and Wei Xing and Dongming Lu},
  title = {{Diverse Image Style Transfer via Invertible Cross-Space Mapping}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01461},
}
Figure