DivSwapper: Towards Diversified Patch-based Arbitrary Style Transfer

Zhizhong Wang, Lei Zhao, Haibo Chen, Zhiwen Zuo, Ailin Li, Wei Xing, Dongming Lu

Abstract

Gram-based and patch-based approaches are two important research lines of style transfer. Recent diversified Gram-based methods have been able to produce multiple and diverse stylized outputs for the same content and style images. However, as another widespread research interest, the diversity of patch-based methods remains challenging due to the stereotyped style swapping process based on nearest patch matching. To resolve this dilemma, in this paper, we dive into the crux of existing patch-based methods and propose a universal and efficient module, termed DivSwapper, for diversified patch-based arbitrary style transfer. The key insight is to use an essential intuition that neural patches with higher activation values could contribute more to diversity. Our DivSwapper is plug-and-play and can be easily integrated into existing patch-based and Gram-based methods to generate diverse results for arbitrary styles. We conduct theoretical analyses and extensive experiments to demonstrate the effectiveness of our method, and compared with state-of-the-art algorithms, it shows superiority in diversity, quality, and efficiency.

Introduction

In the field of Neural Style Transfer (NST), researchers typically follow two primary paths: Gram-based methods, which match global statistics, and patch-based methods, which focus on local textures. While Gram-based methods have evolved to produce diverse outputs for a single style, patch-based methods have remained stubbornly deterministic.

The bottleneck lies in the "style swapping" process. Most patch-based models use Normalized Cross-Correlation (NCC) to find the single "best" matching style patch for every content patch. This rigid "nearest-neighbor" approach creates a stereotyped result; generating multiple versions of a single style often yields identical images. Previous attempts to introduce diversity through noise often resulted in "garbage" outputs where semantic features, such as eyes or mouths, were destroyed.

Methodology: Shifted Style Normalization

DivSwapper is a learning-free, "plug-and-play" module that replaces the standard normalization step in the style-swapping pipeline. In standard implementations, style patches are normalized to a unit length (L2 norm) so that matching depends only on the cosine similarity between patches.

DivSwapper introduces Shifted Style Normalization (SSN). Instead of standard normalization, the module adds a random positive deviation ($\sigma$) to the L2 norm of each style patch. Mathematically, adding this random shift to the denominator reshuffles the distances between patches.

Heatmap illustrating that neural patches with higher activation values (salient features like eyes and nose) are prioritized to contribute more to the diversity of the final output.

Crucially, this shift favors style patches with higher activation values. Because patches with high activations usually contain the most discriminative information—such as edges, salient colors, and semantic details—varying these specific patches leads to significant visual diversity that remains perceptually coherent. This allows for "approximate semantic matching," ensuring that a "sky" patch is still replaced by a "sky-like" style patch, but a different one with each generation.

Experimental Results

The researchers integrated DivSwapper into four major algorithms: CNNMRF, Style-Swap, Avatar-Net, and WCT. Diversity was quantified using $D_{pixel}$ (average pixel distance) and LPIPS (Learned Perceptual Image Patch Similarity).

Example of two distinct, high-quality stylized outputs generated from the same input pair, demonstrating the diversity achieved by the DivSwapper module.

Key Findings

  • Superior Diversity: DivSwapper consistently outperformed the state-of-the-art (SOTA) diversity method, DFP. For example, when applied to the Avatar-Net baseline, DivSwapper achieved an LPIPS score of 0.320, compared to DFP’s 0.264.
  • Visual Quality: Qualitative results show that DivSwapper avoids the "hazy" or "broken" artifacts common in other diversified methods. It even helps alleviate repetitive artifacts, such as the "eye patterns" often seen in rigid semantic-level transfers.
  • Efficiency: Since DivSwapper can be implemented as simple convolutional and deconvolutional layers, it adds negligible computational overhead—often less than 0.01 seconds for feed-forward networks.

Conclusion

DivSwapper provides a robust solution for introducing diversity into patch-based style transfer. By leveraging the intuition that high-activation patches define a style's character, the module generates multiple high-quality versions of a stylized image. While it requires some initial hyperparameter tuning for the sampling range of the random deviation, its efficiency and "plug-and-play" nature make it a powerful tool for the next generation of artistic style transfer models.

Summary updated

Abstract

Gram-based and patch-based approaches are two important research lines of style transfer. Recent diversified Gram-based methods have been able to produce multiple and diverse stylized outputs for the same content and style images. However, as another widespread research interest, the diversity of patch-based methods remains challenging due to the stereotyped style swapping process based on nearest patch matching. To resolve this dilemma, in this paper, we dive into the crux of existing patch-based methods and propose a universal and efficient module, termed DivSwapper, for diversified patch-based arbitrary style transfer. The key insight is to use an essential intuition that neural patches with higher activation values could contribute more to diversity. Our DivSwapper is plug-and-play and can be easily integrated into existing patch-based and Gram-based methods to generate diverse results for arbitrary styles. We conduct theoretical analyses and extensive experiments to demonstrate the effectiveness of our method, and compared with state-of-the-art algorithms, it shows superiority in diversity, quality, and efficiency.
BibTeX
@inproceedings{wang2022divswapper,
  author = {Zhizhong Wang and Lei Zhao and Haibo Chen and Zhiwen Zuo and Ailin Li and Wei Xing and Dongming Lu},
  title = {{DivSwapper: Towards Diversified Patch-based Arbitrary Style Transfer}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=gJfGSpDMJXE},
}
Figure