Adaptive Convolutions for Structure-Aware Style Transfer

Prashanth Chandran, Gaspard Zoss, Paulo F. U. Gotardo, Markus Gross, Derek BradleyCVPR 2021 · OpenReview

Abstract

Style transfer between images is an artistic application of CNNs, where the 'style' of one image is transferred onto another image while preserving the latter's content. The state of the art in neural style transfer is based on Adaptive Instance Normalization (AdaIN), a technique that transfers the statistical properties of style features to a content image, and can transfer a large number of styles in real time. However, AdaIN is a global operation; thus local geometric structures in the style image are often ignored during the transfer. We propose Adaptive Convolutions (AdaConv), a generic extension of AdaIN, to allow for the simultaneous transfer of both statistical and structural styles in real time. Apart from style transfer, our method can also be readily extended to style-based image generation, and other tasks where AdaIN has already been adopted.

Introduction

Neural style transfer has seen rapid advancement, with real-time methods like Adaptive Instance Normalization (AdaIN) becoming the standard for flexible, multi-style synthesis. AdaIN works by aligning the global mean and variance of content features with those of style features. While effective at capturing color palettes and general textures, this statistical approach inherently ignores local geometric structures—such as specific brush stroke shapes or geometric patterns—present in the style image.

Researchers from DisneyResearch|Studios and ETH Zurich have introduced Adaptive Convolutions (AdaConv) to bridge this gap. By predicting full convolution kernels from a style image rather than just global statistics, AdaConv enables the transfer of complex local structures, effectively teaching the network to "paint" with the specific shapes found in a reference style.

Methodology: Adaptive Convolutions (AdaConv)

The core innovation is the AdaConv layer, designed as a generic extension and drop-in replacement for AdaIN.

From Scalars to Kernels

While AdaIN applies a simple linear transformation (scaling and shifting) to features, AdaConv predicts an entire set of convolution kernels and biases based on the style image. This allows the network to have a spatial receptive field capable of recognizing and replicating local geometric patterns. Mathematically, AdaIN can be viewed as a special case of AdaConv where the kernel size is $1 \times 1$. By expanding the kernel size (e.g., to $3 \times 3$), the network gains the ability to replicate complex shapes.

Efficiency via Depthwise-Separable Convolutions

Predicting full 3D convolution kernels for every layer would be computationally prohibitive for real-time applications. To maintain a lightweight architecture, the authors utilize depthwise-separable convolutions. This splits the operation into a spatial component (capturing local structure) and a pointwise component (capturing cross-channel correlations), significantly reducing the parameter count for the kernel prediction networks.

The Prediction Pipeline

The architecture utilizes a VGG-19 encoder to extract features from the style image. A secondary Style Encoder ($E_S$) then compresses these features into a global style descriptor ($W$). Finally, Kernel Prediction Networks ($K$) use this descriptor to generate the adaptive kernels used in the synthesis network. This setup also allows for smooth interpolation in kernel space, enabling transitions between different structural styles.

Results and Evaluation

The authors evaluated AdaConv across general style transfer (using MS-COCO and WikiArt), face stylization, and generative modeling.

Style Transfer Performance

Qualitative comparisons demonstrate that AdaConv successfully captures structural elements that AdaIN misses entirely. For instance, when a style image contains distinct circular patterns or sharp geometric lines, AdaConv replicates these shapes on the content image, whereas AdaIN merely transfers the color and a generic texture.

Qualitative comparison of style transfer results: AdaConv (right) preserves local geometric structures like circles and sharp lines better than AdaIN (middle).

A user study with 185 participants confirmed these findings: 92% of respondents felt AdaConv captured style structure better than AdaIN, and 71.8% preferred the overall results of AdaConv. However, the study also noted a trade-off: AdaIN was slightly better at preserving the original content's integrity (93.9%), suggesting that stronger structural transfer can sometimes distort underlying content.

Style-based Generation

To demonstrate its versatility, AdaConv was integrated into a StyleGAN2-like architecture by replacing weight demodulation with AdaConv layers. The model produced high-quality synthetic images across datasets like FFHQ (faces) and AFHQ (wildlife), proving that AdaConv is a robust, general-purpose building block for generative CNNs.

Examples of style-based generation using AdaConv on face and animal datasets.

Conclusion and Future Work

Adaptive Convolutions provide a powerful extension to the AdaIN framework, enabling structure-aware style transfer without sacrificing real-time performance. By treating style as a learnable spatial filter, AdaConv bridges the gap between statistical texture transfer and structural artistic replication. Future work may involve addressing the content-style trade-off and extending AdaConv to ensure temporal consistency in video stylization applications.

Summary updated

Abstract

Style transfer between images is an artistic application of CNNs, where the 'style' of one image is transferred onto another image while preserving the latter's content. The state of the art in neural style transfer is based on Adaptive Instance Normalization (AdaIN), a technique that transfers the statistical properties of style features to a content image, and can transfer a large number of styles in real time. However, AdaIN is a global operation; thus local geometric structures in the style image are often ignored during the transfer. We propose Adaptive Convolutions (AdaConv), a generic extension of AdaIN, to allow for the simultaneous transfer of both statistical and structural styles in real time. Apart from style transfer, our method can also be readily extended to style-based image generation, and other tasks where AdaIN has already been adopted.
BibTeX
@inproceedings{chandran2021adaptive,
  author = {Prashanth Chandran and Gaspard Zoss and Paulo F. U. Gotardo and Markus Gross and Derek Bradley},
  title = {{Adaptive Convolutions for Structure-Aware Style Transfer}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=F--992jCMPF},
}
Figure