Abstract
Photorealistic style transfer traditionally relies on complex, multi-stage auto-encoder pipelines like PhotoWCT, which are computationally expensive and prone to detail loss. This paper introduces StyleNAS, the first framework to apply Neural Architecture Search (NAS) to this domain. By optimizing for efficiency and perceptual quality, StyleNAS discovers end-to-end architectures that are up to 430x faster than state-of-the-art methods while better preserving fine image details.
Introduction
Photorealistic style transfer aims to apply the "look" of a reference style image to a content image without introducing the painterly distortions common in artistic style transfer. Until now, the gold standard was PhotoWCT, a method that stacks multiple auto-encoders and transformation modules in a non-end-to-end fashion. While effective, PhotoWCT suffers from two major flaws: extreme computational overhead and significant loss of fine image details due to multiple encoding-decoding cycles.
StyleNAS addresses these challenges by systematically applying NAS to generative tasks. The goal is to "search out" simpler, faster, end-to-end architectures that maintain or exceed the quality of hand-designed models.
Methods
The researchers developed a custom NAS approach called StyleNAS, featuring a combinatorial search space of 31 potential operators (VGG-based encoders, WCT modules, various convolution kernels, etc.), resulting in over 2 billion possible architectures.
The search algorithm employs multi-objective optimization, balancing three goals:
- Knowledge Distillation: Minimizing the difference between the searched model and a "Supervisory Oracle" (PhotoWCT).
- Perceptual Loss: Ensuring visual faithfulness to content and style.
- Sparsity/Efficiency: Minimizing active operators to maximize inference speed.
Key Results
StyleNAS identified "surprisingly fast" architectures (StyleNAS-5opt, 7opt, and 9opt) that significantly outperform massive, hand-designed architectures.
- Massive Speedup: On a 768 × 384 image, PhotoWCT took 64.73 seconds, while StyleNAS-5opt took only 0.15 seconds—a speedup of roughly 430x. Even the most complex searched model (StyleNAS-9opt) was nearly 100x faster.
- Superior Quality: StyleNAS models achieved higher Total Variation (TV) scores, indicating much better preservation of sharpness and fine details compared to PhotoWCT.
- Better Style Alignment: Quantitative metrics like FID showed that StyleNAS results were more stylistically consistent with reference images than the PhotoWCT baseline.
Discussion and Future Work
The study proves that the "stacked" approach of previous methods is unnecessary; a single, optimized end-to-end decoder can achieve superior results. Furthermore, the authors demonstrated that NAS can converge for generative tasks, providing a roadmap for other low-level vision problems like denoising or super-resolution.
However, the current method relies on a "Supervisory Oracle" (PhotoWCT) for knowledge distillation, potentially inheriting its biases. Future work will focus on reducing this dependency and extending the framework to artistic style transfer and Generative Adversarial Networks (GANs).
Summary updated