Abstract
Style transfer between images is an artistic application of CNNs, where the 'style' of one image is transferred onto another image while preserving the latter's content. The state of the art in neural style transfer is based on Adaptive Instance Normalization (AdaIN), a technique that transfers the statistical properties of style features to a content image, and can transfer a large number of styles in real time. However, AdaIN is a global operation; thus local geometric structures in the style image are often ignored during the transfer. We propose Adaptive Convolutions (AdaConv), a generic extension of AdaIN, to allow for the simultaneous transfer of both statistical and structural styles in real time. Apart from style transfer, our method can also be readily extended to style-based image generation, and other tasks where AdaIN has already been adopted.
Introduction
Neural style transfer has seen rapid advancement, with real-time methods like Adaptive Instance Normalization (AdaIN) becoming the standard for flexible, multi-style synthesis. AdaIN works by aligning the global mean and variance of content features with those of style features. While effective at capturing color palettes and general textures, this statistical approach inherently ignores local geometric structures—such as specific brush stroke shapes or geometric patterns—present in the style image.
Researchers from DisneyResearch|Studios and ETH Zurich have introduced Adaptive Convolutions (AdaConv) to bridge this gap. By predicting full convolution kernels from a style image rather than just global statistics, AdaConv enables the transfer of complex local structures, effectively teaching the network to "paint" with the specific shapes found in a reference style.
Methodology: Adaptive Convolutions (AdaConv)
The core innovation is the AdaConv layer, designed as a generic extension and drop-in replacement for AdaIN.
From Scalars to Kernels
While AdaIN applies a simple linear transformation (scaling and shifting) to features, AdaConv predicts an entire set of convolution kernels and biases based on the style image. This allows the network to have a spatial receptive field capable of recognizing and replicating local geometric patterns. Mathematically, AdaIN can be viewed as a special case of AdaConv where the kernel size is $1 \times 1$. By expanding the kernel size (e.g., to $3 \times 3$), the network gains the ability to replicate complex shapes.
Efficiency via Depthwise-Separable Convolutions
Predicting full 3D convolution kernels for every layer would be computationally prohibitive for real-time applications. To maintain a lightweight architecture, the authors utilize depthwise-separable convolutions. This splits the operation into a spatial component (capturing local structure) and a pointwise component (capturing cross-channel correlations), significantly reducing the parameter count for the kernel prediction networks.
The Prediction Pipeline
The architecture utilizes a VGG-19 encoder to extract features from the style image. A secondary Style Encoder ($E_S$) then compresses these features into a global style descriptor ($W$). Finally, Kernel Prediction Networks ($K$) use this descriptor to generate the adaptive kernels used in the synthesis network. This setup also allows for smooth interpolation in kernel space, enabling transitions between different structural styles.
Results and Evaluation
The authors evaluated AdaConv across general style transfer (using MS-COCO and WikiArt), face stylization, and generative modeling.
Style Transfer Performance
Qualitative comparisons demonstrate that AdaConv successfully captures structural elements that AdaIN misses entirely. For instance, when a style image contains distinct circular patterns or sharp geometric lines, AdaConv replicates these shapes on the content image, whereas AdaIN merely transfers the color and a generic texture.
A user study with 185 participants confirmed these findings: 92% of respondents felt AdaConv captured style structure better than AdaIN, and 71.8% preferred the overall results of AdaConv. However, the study also noted a trade-off: AdaIN was slightly better at preserving the original content's integrity (93.9%), suggesting that stronger structural transfer can sometimes distort underlying content.
Style-based Generation
To demonstrate its versatility, AdaConv was integrated into a StyleGAN2-like architecture by replacing weight demodulation with AdaConv layers. The model produced high-quality synthetic images across datasets like FFHQ (faces) and AFHQ (wildlife), proving that AdaConv is a robust, general-purpose building block for generative CNNs.
Conclusion and Future Work
Adaptive Convolutions provide a powerful extension to the AdaIN framework, enabling structure-aware style transfer without sacrificing real-time performance. By treating style as a learnable spatial filter, AdaConv bridges the gap between statistical texture transfer and structural artistic replication. Future work may involve addressing the content-style trade-off and extending AdaConv to ensure temporal consistency in video stylization applications.
Summary updated