Pair-wise Exchangeable Feature Extraction for Arbitrary Style Transfer

Zhijie Wu, Chunjin Song, Yang Zhou, Minglun Gong, Hui Huang

Abstract

Arbitrary style transfer aims to synthesize images that combine the content of one image with the style of another. Existing methods often suffer from a "compatibility gap" where style features are forced onto content without considering their structural relationship, leading to visual artifacts. We present EFANet (Exchangeable Feature Alignment Network), which utilizes a pair-wise feature exchange mechanism to align common characteristics between content and style. By incorporating a novel whitening loss to purify content features, EFANet produces high-quality, artifact-free stylizations in real-time.

Introduction

Style transfer is the process of applying the artistic texture and color of a "style" image to the structural layout of a "content" image. While early optimization-based methods produced high-quality results, they were computationally expensive. Modern arbitrary style transfer models, such as AdaIN and WCT, offer real-time performance but often struggle with a "compatibility gap." These models typically treat style as a fixed set of statistics to be imposed on the content, frequently ignoring semantic and structural discrepancies. This mismatch results in visual artifacts, such as distorted patterns, messy textures, or color leakage that disregards content boundaries. EFANet addresses this by proposing that style features must be compatible with content information for a natural-looking transfer.

Methodology

The core innovation of EFANet is the transition from one-way statistical matching to joint pair-wise analysis through several key components:

Feature Exchange Block (FEB)

Inspired by domain adaptation, the FEB module analyzes style vectors from both the content and style images. It disentangles these into "common" and "unique" features. By identifying shared characteristics, the model can align style features to better fit the specific geometry and structure of the content image.

Exchangeable Style Features

Rather than using raw style features, EFANet employs these aligned, "exchangeable" features. This adaptation allows the style patterns to conform to the content's underlying structure, reducing the likelihood of semantic distortion.

Whitening Loss ($L_{whi}$)

To prepare the content image for restyling, EFANet introduces a Whitening Loss. This loss "purifies" the content features by decorrelating feature channels, effectively removing the original style patterns. This creates a "blank canvas" that is more receptive to the target style, preventing clashes between the original and target textures.

Multi-scale Architecture

The framework utilizes a pre-trained VGG-19 encoder and a corresponding decoder. EFANet modules are integrated at multiple layers (from coarse to fine) to ensure the capture of both global color schemes and intricate local brushstrokes.

Experimental Setup

The model was trained using standard large-scale datasets:

  • Content Images: The Place365 database, providing a diverse range of real-world scenes.
  • Style Images: The WikiArt dataset, containing a vast collection of artistic styles.
  • Evaluation: Quantitative testing involved 100 random image pairs. A user study was also conducted with 30 subjects evaluating 900 stylization results generated from 30 content and 30 style images.

Results and Discussion

EFANet demonstrates significant improvements over existing real-time methods in both qualitative and quantitative measures.

Comparison of stylization results between AdaIN and EFANet variants.

Visual Quality and Structure Preservation

As shown in Figure 1, EFANet preserves semantic structures—such as the distinct lines of a building—far better than baselines like AdaIN. While AdaIN often "smears" textures across important features, EFANet maintains structural integrity while applying rich artistic patterns. The ablation study shows that the full EFANet model (Ours) achieves better texture consistency than versions without the common feature (CF) alignment.

Quantitative Performance and User Preference

Quantitatively, EFANet achieved the lowest perceptual loss among compared feed-forward models. It showed significantly lower style loss, indicating a more faithful capture of artistic intent. In a blind user study, EFANet was preferred as the "best stylization" 20% of the time, outperforming WCT (15.5%) and AdaIN (11%).

Computational Efficiency

Despite its improved quality, EFANet remains highly efficient. It runs at approximately 0.023 seconds per image on a Titan V GPU, making it nearly 20 times faster than WCT and comparable to the fastest real-time methods like AdaIN (0.019s).

Conclusion

EFANet introduces a paradigm shift in arbitrary style transfer by treating the content-style pair as a relationship rather than a forced overlay. The introduction of the Whitening Loss and the Feature Exchange Block allows for cleaner, more structured artistic results. While the model favors "fully stylized" results which may occasionally sacrifice fine content details, it represents a significant step toward artifact-free, real-time artistic synthesis. Future work may involve automating spatial control via semantic segmentation to further enhance user flexibility.

Summary updated

BibTeX
@inproceedings{wu2020pair,
  author = {Zhijie Wu and Chunjin Song and Yang Zhou and Minglun Gong and Hui Huang},
  title = {{Pair-wise Exchangeable Feature Extraction for Arbitrary Style Transfer}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6914},
}
Figure