Abstract
Gram-based and patch-based approaches are two important research lines of style transfer. Recent diversified Gram-based methods have been able to produce multiple and diverse stylized outputs for the same content and style images. However, as another widespread research interest, the diversity of patch-based methods remains challenging due to the stereotyped style swapping process based on nearest patch matching. To resolve this dilemma, in this paper, we dive into the crux of existing patch-based methods and propose a universal and efficient module, termed DivSwapper, for diversified patch-based arbitrary style transfer. The key insight is to use an essential intuition that neural patches with higher activation values could contribute more to diversity. Our DivSwapper is plug-and-play and can be easily integrated into existing patch-based and Gram-based methods to generate diverse results for arbitrary styles. We conduct theoretical analyses and extensive experiments to demonstrate the effectiveness of our method, and compared with state-of-the-art algorithms, it shows superiority in diversity, quality, and efficiency.
Introduction
In the field of Neural Style Transfer (NST), researchers typically follow two primary paths: Gram-based methods, which match global statistics, and patch-based methods, which focus on local textures. While Gram-based methods have evolved to produce diverse outputs for a single style, patch-based methods have remained stubbornly deterministic.
The bottleneck lies in the "style swapping" process. Most patch-based models use Normalized Cross-Correlation (NCC) to find the single "best" matching style patch for every content patch. This rigid "nearest-neighbor" approach creates a stereotyped result; generating multiple versions of a single style often yields identical images. Previous attempts to introduce diversity through noise often resulted in "garbage" outputs where semantic features, such as eyes or mouths, were destroyed.
Methodology: Shifted Style Normalization
DivSwapper is a learning-free, "plug-and-play" module that replaces the standard normalization step in the style-swapping pipeline. In standard implementations, style patches are normalized to a unit length (L2 norm) so that matching depends only on the cosine similarity between patches.
DivSwapper introduces Shifted Style Normalization (SSN). Instead of standard normalization, the module adds a random positive deviation ($\sigma$) to the L2 norm of each style patch. Mathematically, adding this random shift to the denominator reshuffles the distances between patches.
Crucially, this shift favors style patches with higher activation values. Because patches with high activations usually contain the most discriminative information—such as edges, salient colors, and semantic details—varying these specific patches leads to significant visual diversity that remains perceptually coherent. This allows for "approximate semantic matching," ensuring that a "sky" patch is still replaced by a "sky-like" style patch, but a different one with each generation.
Experimental Results
The researchers integrated DivSwapper into four major algorithms: CNNMRF, Style-Swap, Avatar-Net, and WCT. Diversity was quantified using $D_{pixel}$ (average pixel distance) and LPIPS (Learned Perceptual Image Patch Similarity).
Key Findings
- Superior Diversity: DivSwapper consistently outperformed the state-of-the-art (SOTA) diversity method, DFP. For example, when applied to the Avatar-Net baseline, DivSwapper achieved an LPIPS score of 0.320, compared to DFP’s 0.264.
- Visual Quality: Qualitative results show that DivSwapper avoids the "hazy" or "broken" artifacts common in other diversified methods. It even helps alleviate repetitive artifacts, such as the "eye patterns" often seen in rigid semantic-level transfers.
- Efficiency: Since DivSwapper can be implemented as simple convolutional and deconvolutional layers, it adds negligible computational overhead—often less than 0.01 seconds for feed-forward networks.
Conclusion
DivSwapper provides a robust solution for introducing diversity into patch-based style transfer. By leveraging the intuition that high-activation patches define a style's character, the module generates multiple high-quality versions of a stylized image. While it requires some initial hyperparameter tuning for the sampling range of the random deviation, its efficiency and "plug-and-play" nature make it a powerful tool for the next generation of artistic style transfer models.
Summary updated