Playing Lottery Tickets in Style Transfer Models

Meihao Kong, Jing Huo, Wenbin Li, Jing Wu, Yu-Kun Lai, Yang Gao

Abstract

Style transfer has achieved great success and attracted a wide range of attention from both academic and industrial communities due to its flexible application scenarios. However, the dependence on a pretty large VGG-based autoencoder leads to existing style transfer models having high parameter complexities, which limits their applications on resource-constrained devices. Compared with many other tasks, the compression of style transfer models has been less explored. Recently, the lottery ticket hypothesis (LTH) has shown great potential in finding extremely sparse matching subnetworks which can achieve on par or even better performance than the original full networks when trained in isolation. In this work, we for the first time perform an empirical study to verify whether such trainable matching subnetworks also exist in style transfer models. Specifically, we take two most popular style transfer models, i.e., AdaIN and SANet, as the main testbeds, which represent global and local transformation based style transfer methods respectively. We carry out extensive experiments and comprehensive analysis, and draw the following conclusions. (1) Compared with fixing the VGG encoder, style transfer models can benefit more from training the whole network together. (2) Using iterative magnitude pruning, we find the matching subnetworks at 89.2% sparsity in AdaIN and 73.7% sparsity in SANet, which demonstrates that style transfer models can play lottery tickets too. (3) The feature transformation module should also be pruned to obtain a much sparser model without affecting the existence and quality of the matching subnetworks. (4) Besides AdaIN and SANet, other models such as LST, MANet, AdaAttN and MCCNet can also play lottery tickets, which shows that LTH can be generalized to various style transfer models.

Introduction

Neural Style Transfer (NST) has become a staple of creative AI, but it comes with a heavy computational tax. Most state-of-the-art models rely on a massive, pre-trained VGG-19 backbone as an encoder. Because these encoders are typically kept "frozen" during training, the resulting models are parameter-heavy and difficult to deploy on mobile phones or edge devices. This paper demonstrates that the Lottery Ticket Hypothesis (LTH) holds true for NST models, revealing that these typically bloated architectures contain tiny, sparse subnetworks ("winning tickets") capable of matching or exceeding the performance of the full model.

Methodology: Iterative Magnitude Pruning in NST

The authors primarily investigated two representative models: AdaIN (global transformation) and SANet (local transformation). Their approach involved several key technical shifts:

  1. Iterative Magnitude Pruning (IMP): Instead of pruning all at once, they used an iterative process. They trained the model, pruned the 20% of weights with the smallest magnitudes, reset the remaining weights to their original values, and repeated the process until reaching the desired sparsity.
  2. The "Plus" Strategy: Traditionally, NST models fix the VGG encoder. The authors found that training the entire network (encoder + transformation module + decoder) together significantly improved performance and made it easier to find sparse winning tickets.
  3. Pruning the Transformation Module: Unlike previous works that only pruned the autoencoder, this study pruned the learnable feature transformation modules (like the attention mechanism in SANet), which often contain a significant portion of the model's parameters.
The Iterative Magnitude Pruning (IMP) pipeline applied to style transfer models.

Experimental Results

The most significant finding is that LTH is universal across style transfer architectures. Whether the model uses global statistics (AdaIN) or local attention (SANet), winning tickets exist.

  • Extreme Sparsity: The authors identified winning tickets at 89.2% sparsity for AdaIN+ and 73.7% sparsity for SANet+ that matched or beat the full model's performance.
  • Quantitative Edge: In AdaIN+, the winning ticket at 59% sparsity actually achieved a lower average error (5.134) than the full model (5.685).
  • Generalization: The findings extended to other models like LST, MANet, AdaAttN, and MCCNet. Notably, LST yielded a winning ticket with only 6.9% of its original parameters.
Average test error vs. remaining weights for AdaIN+ and SANet+ models, showing the 'winning ticket' dip where sparse models outperform the dense baseline.

Conclusion

By unfreezing and pruning the VGG encoder, the authors showed that the standard practice of using a fixed, massive pre-trained backbone is highly inefficient for this specific task. Surprisingly, extremely sparse models sometimes produced better visual results with fewer artifacts than the full model, likely due to reduced overfitting. While the current pruning is unstructured, these findings offer a clear path toward high-quality style transfer on resource-constrained devices. Future work will focus on hardware acceleration and broader applications like video stylization.

Summary updated

Abstract

Style transfer has achieved great success and attracted a wide range of attention from both academic and industrial communities due to its flexible application scenarios. However, the dependence on a pretty large VGG-based autoencoder leads to existing style transfer models having high parameter complexities, which limits their applications on resource-constrained devices. Compared with many other tasks, the compression of style transfer models has been less explored. Recently, the lottery ticket hypothesis (LTH) has shown great potential in finding extremely sparse matching subnetworks which can achieve on par or even better performance than the original full networks when trained in isolation. In this work, we for the first time perform an empirical study to verify whether such trainable matching subnetworks also exist in style transfer models. Specifically, we take two most popular style transfer models, i.e., AdaIN and SANet, as the main testbeds, which represent global and local transformation based style transfer methods respectively. We carry out extensive experiments and comprehensive analysis, and draw the following conclusions. (1) Compared with fixing the VGG encoder, style transfer models can benefit more from training the whole network together. (2) Using iterative magnitude pruning, we find the matching subnetworks at 89.2% sparsity in AdaIN and 73.7% sparsity in SANet, which demonstrates that style transfer models can play lottery tickets too. (3) The feature transformation module should also be pruned to obtain a much sparser model without affecting the existence and quality of the matching subnetworks. (4) Besides AdaIN and SANet, other models such as LST, MANet, AdaAttN and MCCNet can also play lottery tickets, which shows that LTH can be generalized to various style transfer models.
BibTeX
@misc{kong2022playing,
  author = {Meihao Kong and Jing Huo and Wenbin Li and Jing Wu and Yu-Kun Lai and Yang Gao},
  title = {{Playing Lottery Tickets in Style Transfer Models}},
  year = {2022},
  eprint = {2203.13802},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.13802},
}
Figure