Diversified Patch-based Style Transfer with Shifted Style Normalization

Zhizhong Wang, Lei Zhao, Haibo Chen, Zhiwen Zuo, Ailin Li, Wei Xing, Dongming Lu

Introduction

Neural Style Transfer (NST) has traditionally been divided into two paradigms: global statistics matching (Gram-based) and local texture swapping (patch-based). While Gram-based methods have evolved to produce diverse outputs, patch-based methods often suffer from deterministic "stereotyping." Because these methods typically rely on Normalized Cross-Correlation (NCC) to find the single best match for each content patch, they produce identical results for any given content-style pair. This lack of variation limits the creative potential of patch-based algorithms.

To address this, we present DivSwapper, a universal, learning-free module designed to introduce diversity into patch-based style transfer without compromising semantic integrity or requiring retraining.

Methodology: Shifted Style Normalization (SSN)

The core of DivSwapper is Shifted Style Normalization (SSN). In standard patch-based matching, style patches are normalized by their L2 norm. DivSwapper introduces a random positive deviation, $\sigma$, to this denominator:

$$\hat{\phi}_j(F_s) = \frac{\phi_j(F_s)}{\|\phi_j(F_s)\| + \sigma}$$

By perturbing the normalization factor, SSN shifts the "nearest neighbor" selection process. This allows the algorithm to select alternative, high-quality style patches that are semantically similar but visually distinct from the original deterministic choice.

The DivSwapper Workflow

  1. Feature Extraction: Neural patches are extracted from content and style images using a pre-trained VGG19 backbone.
  2. SSN Application: Style patches undergo Shifted Style Normalization.
  3. Patch Matching: Shifted patches act as filters in a convolutional layer to identify new nearest neighbors via NCC.
  4. Reconstruction: Content patches are swapped with the original (unshifted) style patches corresponding to the new matches, then passed through a decoder to generate the final image.

Experimental Results

The effectiveness of DivSwapper was evaluated using 36 content-style pairs, generating over 6,000 images to measure diversity via Average Pixel Distance ($D_{pixel}$) and Learned Perceptual Image Patch Similarity (LPIPS).

Figure 4: Qualitative results showing the diversity achieved by DivSwapper when integrated with CNNMRF and Avatar-Net baselines.

Diversity and Quality

As shown in Figure 4, DivSwapper enables existing baselines like CNNMRF and Avatar-Net to generate multiple distinct artistic interpretations. Unlike previous perturbation methods that often introduced "hazy" artifacts or structural failures, DivSwapper maintains high visual quality. It effectively varies colors and textures—such as the sky in the cityscape or the feather patterns on the hawk—while preserving the recognizable structure of the content image.

Performance and User Preference

DivSwapper is highly efficient, adding negligible computational overhead (e.g., an increase of only 0.012s when applied to Avatar-Net). In a user study involving 20 participants, DivSwapper's outputs were preferred over both the original baselines and other state-of-the-art diversified methods, receiving approximately 39% of the total votes.

Conclusion and Future Work

DivSwapper provides a simple yet powerful "plug-and-play" solution for diversifying patch-based style transfer. By leveraging the intuition that high-activation patches drive visual diversity, SSN allows for meaningful variation without the need for complex retraining. Future research will focus on optimizing the sampling range of $\sigma$ for different architectures and exploring more constrained diversity for tasks requiring high semantic precision, such as portrait-to-portrait transfer.

Summary updated

BibTeX
@inproceedings{wang2022diversified,
  author = {Zhizhong Wang and Lei Zhao and Haibo Chen and Zhiwen Zuo and Ailin Li and Wei Xing and Dongming Lu},
  title = {{Diversified Patch-based Style Transfer with Shifted Style Normalization}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=gJfGSpDMJXE},
}
Figure