Line Search-Based Feature Transformation for Fast, Stable, and Tunable Content-Style Control in Photorealistic Style Transfer

Tai-Yin Chiu, Danna GurariWACV 2023

TL;DR

This paper introduces LS-FT, a novel feature transformation for photorealistic style transfer that offers a better balance between preserving content and applying style compared to existing methods. It achieves this through a combination of iterative optimization with line search and a modified iterative approach, resulting in faster, more stable, and tunable stylization across various models.

What Problem Does This Paper Address?

Photorealistic style transfer aims to make an image look like it was created in the style of another image, while retaining the original content. Modern approaches often use autoencoders with feature transformations to achieve this. However, a key challenge is finding a transformation that works well across different autoencoder architectures and allows users to control the trade-off between preserving the original image's content and the intensity of the applied style. Existing methods often fail to generalize, produce artifacts, or lack fine-grained control over this balance. Some methods are also computationally expensive or unstable.

Methods

The paper proposes a new transformation called LS-FT (Line Search-Based Feature Transformation). It builds upon a previous method called Iterative Feature Transformation (IterFT), which offered tunable content-style balance but suffered from instability and slowness.

LS-FT introduces two main improvements:

  1. Modified Iterative Feature Transformation: This involves adding centralization (subtracting the mean) before and decentralization (adding back the mean) after the core feature transformation step. This modification is inspired by the observation that other successful transformations consistently use these steps, and it leads to more stable and reasonable results compared to the original IterFT.
  2. Line Search-Based Optimization: To address the slowness of IterFT (which requires multiple iterations), LS-FT incorporates a line search to find the optimal step size (learning rate) for updating the feature in a single iteration. This is achieved by solving a cubic equation derived from minimizing the loss function. This allows for a significant speedup, as only one iteration is often sufficient.

Datasets

The paper uses the PST dataset for evaluation. This dataset consists of 786 pairs of content and style images, making it a substantial benchmark for photorealistic style transfer.

Key Insights and Ideas

  • Generalization is a major issue: Existing feature transformations don't perform consistently across different photorealistic style transfer architectures.
  • Centralization/Decentralization matters: Adding these simple pre- and post-processing steps to the core feature transformation significantly improves stability and result quality.
  • Line search for speed: Optimizing the update step with line search dramatically reduces the number of iterations needed, leading to substantial speedups without sacrificing quality.
  • Tunability is key: The proposed LS-FT offers a tunable parameter that allows users to explicitly control the balance between content preservation and style intensity.

Key Results

  • Improved Content-Style Balance: LS-FT consistently achieves a better balance between content preservation and style transfer strength across various models (WCT2, PhotoWCT, PhotoWCT2, PCA-d) compared to existing methods like AdaIN, ZCA, OST, and MAST.
  • Faster Performance: LS-FT is significantly faster than its predecessor, Modified IterFT, achieving stylization 7-8x faster. It is also competitive with or faster than ZCA and OST.
  • Stability: The modified iterative approach with centralization/decentralization leads to stable and reasonable results, unlike the original IterFT which often produced artifacts.
  • Qualitative Improvements: Visual examples show LS-FT producing more aesthetically pleasing results with better content preservation and stronger style transfer compared to baselines. For instance, it can boost stylization for WCT2 and improve content preservation for PhotoWCT and PCA-d.

Limitations and Future Work

  • Parameter Tuning: While LS-FT offers tunable parameters (like $\alpha$ for content-style balance), finding the optimal values might still require some experimentation.
  • Computational Cost of Line Search: Although LS-FT is faster overall, the line search itself involves solving a cubic equation, which adds some computational overhead compared to simpler, non-iterative methods like AdaIN. However, the paper argues this is a worthwhile trade-off for the improved quality and control.
  • Future Work: The paper doesn't explicitly detail future work, but the success of LS-FT suggests potential for further exploration in optimizing the line search process, exploring different regularization techniques, and applying the core ideas to other image generation tasks.

Summary updated

BibTeX
@inproceedings{chiu2023line,
  author = {Tai-Yin Chiu and Danna Gurari},
  title = {{Line Search-Based Feature Transformation for Fast, Stable, and Tunable Content-Style Control in Photorealistic Style Transfer}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure