Exploiting Spatial Relation for Reducing Distortion in Style Transfer

Jia-Ren Chang, Yong-Sheng ChenWACV 2021

TL;DR

This paper introduces SRVGG, a modified VGG network with a spatial relation module, and a novel relation loss to combat spatial distortions in neural style transfer. By explicitly modeling long-range spatial dependencies, their method significantly improves image consistency and visual appeal without requiring segmentation masks.

What Problem Does This Paper Address?

Neural style transfer, a technique that allows users to repaint an image in the style of another, has seen remarkable progress thanks to deep convolutional neural networks (CNNs). However, a persistent challenge has been the generation of spatially inconsistent results, often manifesting as noticeable artifacts. Existing solutions, such as using segmentation masks or affinity-based image matting, can be computationally expensive and require additional manual input. This paper aims to address this gap by developing a method that inherently preserves spatial information within the style transfer process, leading to more visually coherent and artifact-free stylized images.

Methods

The core innovation of this paper lies in two components:

  • Spatial Relation-Augmented VGG (SRVGG): The authors modify the standard VGG-19 network by incorporating a "spatial relation module" after specific convolutional layers (conv4_1 and conv5_1). This module is designed to capture long-range spatial dependencies between different locations in an image. It computes a relation matrix that represents how features at one spatial position relate to features at all other positions. This module can be viewed as a form of non-local operation.
  • Relation Loss: To leverage the spatial information captured by SRVGG, a new "relation loss" is introduced. This loss function encourages the spatial relationships in the stylized output image to be similar to those in the original content image. By minimizing this loss alongside traditional content and style losses, the method effectively constrains the stylization process to preserve spatial structure.

The proposed SRVGG network is trained on ImageNet for image recognition tasks, demonstrating improved accuracy. Subsequently, this trained SRVGG is used as a feature extractor for both optimization-based and feedforward-based style transfer methods.

Datasets

  • ImageNet: Used for training the SRVGG network for image recognition.
  • WikiArt: Used as style images for style transfer experiments.
  • MS-COCO: Used as content images for style transfer experiments.

The paper does not introduce new benchmarks but evaluates its method against existing datasets and tasks.

Key Insights and Ideas

  • Spatial Dependencies are Key: The paper strongly suggests that explicitly modeling long-range spatial relationships within the feature extraction process is crucial for mitigating artifacts in style transfer.
  • SRVGG as a Better Feature Extractor: By augmenting VGG with spatial relation modules, the authors show that SRVGG can capture richer, spatially aware features that are beneficial for style transfer.
  • Relation Loss as a Constraint: The novel relation loss acts as an effective constraint, guiding the style transfer process to maintain the structural integrity of the content image without explicit segmentation.
  • Universality: The proposed framework is shown to be applicable to both optimization-based and feedforward-based style transfer methods, highlighting its general utility.

Key Results

  • Qualitative Improvements: The proposed method consistently generates stylized images with significantly reduced spatial distortion and artifacts compared to baseline methods like Neural Style Transfer [4], AdaIN [8], and WCT [11]. This is visually evident in the provided figures, where details like rose windows and facial features are preserved.
  • Quantitative Improvements: In a user study, the proposed method (specifically the modified AdaIN version) was rated as the most visually appealing and spatially consistent by participants. It also achieved the lowest relation loss, content loss, and style loss in quantitative evaluations.
  • Efficiency: For feedforward methods, the proposed approach maintains efficiency comparable to existing methods like AdaIN, while offering superior spatial consistency.

Limitations and Future Work

  • Computational Overhead: While the SRVGG module is designed to capture long-range dependencies, it does introduce some additional computational cost compared to a standard VGG network. The paper notes a slight increase in computational overhead.
  • Hyperparameter Tuning: The effectiveness of the relation loss is dependent on its weight (γ). While the paper explores different values, finding the optimal balance for diverse content and style combinations might require further investigation.
  • Future Work: The authors suggest that SRVGG's ability to capture long-range spatial dependency could be beneficial for other computer vision tasks beyond style transfer, such as image generation and object detection. Further exploration of its application in these areas is a promising direction.

Summary updated

BibTeX
@inproceedings{chang2021exploiting,
  author = {Jia-Ren Chang and Yong-Sheng Chen},
  title = {{Exploiting Spatial Relation for Reducing Distortion in Style Transfer}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure