StyleNAS: An Empirical Study of Neural Architecture Search to Uncover Surprisingly Fast End-to-End Universal Style Transfer Networks

Jie An, Haoyi Xiong, Jinwen Ma, Jiebo Luo, Jun HuanarXiv 2019 · arXiv

Abstract

Photorealistic style transfer traditionally relies on complex, multi-stage auto-encoder pipelines like PhotoWCT, which are computationally expensive and prone to detail loss. This paper introduces StyleNAS, the first framework to apply Neural Architecture Search (NAS) to this domain. By optimizing for efficiency and perceptual quality, StyleNAS discovers end-to-end architectures that are up to 430x faster than state-of-the-art methods while better preserving fine image details.

Introduction

Photorealistic style transfer aims to apply the "look" of a reference style image to a content image without introducing the painterly distortions common in artistic style transfer. Until now, the gold standard was PhotoWCT, a method that stacks multiple auto-encoders and transformation modules in a non-end-to-end fashion. While effective, PhotoWCT suffers from two major flaws: extreme computational overhead and significant loss of fine image details due to multiple encoding-decoding cycles.

StyleNAS addresses these challenges by systematically applying NAS to generative tasks. The goal is to "search out" simpler, faster, end-to-end architectures that maintain or exceed the quality of hand-designed models.

Methods

The researchers developed a custom NAS approach called StyleNAS, featuring a combinatorial search space of 31 potential operators (VGG-based encoders, WCT modules, various convolution kernels, etc.), resulting in over 2 billion possible architectures.

The search algorithm employs multi-objective optimization, balancing three goals:

  1. Knowledge Distillation: Minimizing the difference between the searched model and a "Supervisory Oracle" (PhotoWCT).
  2. Perceptual Loss: Ensuring visual faithfulness to content and style.
  3. Sparsity/Efficiency: Minimizing active operators to maximize inference speed.
Comparison of the stacked PhotoWCT architecture (a) against the efficient, end-to-end StyleNAS variants (b, c, d).

Key Results

StyleNAS identified "surprisingly fast" architectures (StyleNAS-5opt, 7opt, and 9opt) that significantly outperform massive, hand-designed architectures.

  • Massive Speedup: On a 768 × 384 image, PhotoWCT took 64.73 seconds, while StyleNAS-5opt took only 0.15 seconds—a speedup of roughly 430x. Even the most complex searched model (StyleNAS-9opt) was nearly 100x faster.
  • Superior Quality: StyleNAS models achieved higher Total Variation (TV) scores, indicating much better preservation of sharpness and fine details compared to PhotoWCT.
  • Better Style Alignment: Quantitative metrics like FID showed that StyleNAS results were more stylistically consistent with reference images than the PhotoWCT baseline.
A visual result from StyleNAS-7opt demonstrating high-quality style transfer with an inference time of only 0.18s.

Discussion and Future Work

The study proves that the "stacked" approach of previous methods is unnecessary; a single, optimized end-to-end decoder can achieve superior results. Furthermore, the authors demonstrated that NAS can converge for generative tasks, providing a roadmap for other low-level vision problems like denoising or super-resolution.

However, the current method relies on a "Supervisory Oracle" (PhotoWCT) for knowledge distillation, potentially inheriting its biases. Future work will focus on reducing this dependency and extending the framework to artistic style transfer and Generative Adversarial Networks (GANs).

Summary updated

Abstract

Neural Architecture Search (NAS) has been widely studied for designing discriminative deep learning models such as image classification, object detection, and semantic segmentation. As a large number of priors have been obtained through the manual design of architectures in the fields, NAS is usually considered as a supplement approach. In this paper, we have significantly expanded the application areas of NAS by performing an empirical study of NAS to search generative models, or specifically, auto-encoder based universal style transfer, which lacks systematic exploration, if any, from the architecture search aspect. In our work, we first designed a search space where common operators for image style transfer such as VGG-based encoders, whitening and coloring transforms (WCT), convolution kernels, instance normalization operators, and skip connections were searched in a combinatorial approach. With a simple yet effective parallel evolutionary NAS algorithm with multiple objectives, we derived the first group of end-to-end deep networks for universal photorealistic style transfer. Comparing to random search, a NAS method that is gaining popularity recently, we demonstrated that carefully designed search strategy leads to much better architecture design. Finally compared to existing universal style transfer networks for photorealistic rendering such as PhotoWCT that stacks multiple well-trained auto-encoders and WCT transforms in a non-end-to-end manner, the architectures designed by StyleNAS produce better style-transferred images with details preserving, using a tiny number of operators/parameters, and enjoying around 500x inference time speed-up.
BibTeX
@misc{an2019stylenas,
  author = {Jie An and Haoyi Xiong and Jinwen Ma and Jiebo Luo and Jun Huan},
  title = {{StyleNAS: An Empirical Study of Neural Architecture Search to Uncover Surprisingly Fast End-to-End Universal Style Transfer Networks}},
  year = {2019},
  eprint = {1906.02470},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.02470},
}
Figure