Abstract
An unprecedented booming has been witnessed in the research area of artistic style transfer ever since Gatys et al. introduced the neural method. One of the remaining challenges is to balance a trade-off among three critical aspects—speed, flexibility, and quality: (i) the vanilla optimization-based algorithm produces impressive results for arbitrary styles, but is unsatisfyingly slow due to its iterative nature, (ii) the fast approximation methods based on feed-forward neural networks generate satisfactory artistic effects but bound to only a limited number of styles, and (iii) feature-matching methods like AdaIN achieve arbitrary style transfer in a real-time manner but at a cost of the compromised quality. We find it considerably difficult to balance the trade-off well merely using a single feed-forward step and ask, instead, whether there exists an algorithm that could adapt quickly to any style, while the adapted model maintains high efficiency and good image quality. Motivated by this idea, we propose a novel method, coined MetaStyle, which formulates the neural style transfer as a bilevel optimization problem and combines learning with only a few post-processing update steps to adapt to a fast approximation model with satisfying artistic effects, comparable to the optimization-based methods for an arbitrary style. The qualitative and quantitative analysis in the experiments demonstrates that the proposed approach achieves high-quality arbitrary artistic style transfer effectively, with a good trade-off among speed, flexibility, and quality.
Introduction
Since the introduction of Neural Style Transfer (NST), the field has struggled with an "impossible trinity" of competing objectives: speed, flexibility, and quality. Traditional optimization-based methods offer high quality and flexibility but are too slow for real-time use. Conversely, feed-forward networks provide speed but are restricted to a fixed set of styles. Recent arbitrary style transfer methods like AdaIN offer flexibility and speed but often at the expense of visual quality, introducing artifacts or structural distortions. MetaStyle addresses this by framing style transfer as a meta-learning problem, enabling rapid adaptation to any style while maintaining the performance of specialized networks.
Methodology: Bilevel Optimization for Style Transfer
The core innovation of MetaStyle is the use of a bilevel optimization framework, inspired by meta-learning algorithms like MAML. Instead of training a network to perform a specific style transfer, the goal is to learn a "style-neutral" initialization that can be quickly specialized.
The training process consists of two nested loops:
- Inner Loop (Adaptation): For a given style, the model performs a small number of gradient descent steps to minimize a perceptual loss function (combining content and style). This specializes the model to the target artwork.
- Outer Loop (Meta-Training): The algorithm optimizes the initial weights of the model so that the inner loop adaptation is maximally effective across a diverse range of styles.
By training on the MS-COCO dataset for content and the WikiArt dataset for styles, the model develops a robust understanding of image structures. This "blank slate" representation allows for high-quality results with minimal post-processing.
Experimental Results
MetaStyle effectively balances the three critical aspects of NST. Once adapted to a specific style—a process taking approximately 24 seconds for 200 iterations—the model performs inference in just 0.0145 seconds for a 512px image, matching the speed of real-time feed-forward networks.
Qualitatively, MetaStyle outperforms existing arbitrary transfer methods by avoiding common artifacts like deformed color blobs or unnatural cracks. It maintains the structural integrity of the content image while applying artistic strokes comparable to slow, iterative optimization methods. Furthermore, the model is highly efficient, utilizing only 1.68 million parameters, and supports smooth interpolation between styles through parameter blending.
Conclusion
MetaStyle successfully navigates the trade-off between speed, flexibility, and quality in neural style transfer. By leveraging meta-learning, it provides a path toward high-quality, arbitrary style transfer that remains computationally efficient. While the initial adaptation phase introduces a minor overhead compared to zero-shot methods, the resulting quality and real-time inference capabilities represent a significant advancement in the field. Future work may focus on automating hyperparameter tuning and enhancing temporal stability for video applications.
Summary updated