Consistent Video Style Transfer via Compound Regularization

Wenjing Wang, Jizheng Xu, Li Zhang, Yue Wang, Jiaying LiuAAAI 2020 · DOI

Abstract

Video style transfer aims to apply artistic styles to video sequences while maintaining temporal consistency. This paper presents a zero-shot framework that addresses the common issue of flickering artifacts through a novel compound regularization term and sequence-level feature sharing. Our approach achieves high stability and artistic quality without the computational burden of iterative optimization or the fragility of per-frame optical flow estimation.

Introduction

Neural Style Transfer (NST) has seen significant success in static imagery, yet its extension to video remains challenging due to temporal instability. Frame-by-frame application of NST often results in distracting flickering because the algorithm interprets each frame slightly differently. Existing solutions typically fall into two camps: slow optimization-based methods or feed-forward models that use pre-computed optical flow to enforce consistency. However, optical flow is prone to "ghosting" artifacts and often fails to provide long-term stability.

This work introduces a fast, feed-forward model designed for mathematical stability. By moving beyond a heavy reliance on brittle flow data during inference, we provide a robust solution for stylizing videos with intense motion and illumination changes.

Methodology

The proposed framework introduces three key innovations to balance stylization quality with temporal stability.

Compound Regularization ($L_{comp}$)

We model temporal consistency as a mapping problem, deriving a loss function that accounts for both local jitter ($\Delta$) and global motion ($W$). By training the network to be invariant to these combined transformations, the model learns to produce stable outputs even in the presence of input noise or rapid movement.

Original content frame with style reference inset.

Sequence-Level Global Feature Sharing

Standard style transfer methods often use per-frame statistics (mean and variance) to define style. This leads to "style drift" when new objects enter the frame. We address this by calculating average style features across the entire video sequence (or a large chunk of it) and applying these global statistics consistently to every frame.

Dynamic Inter-Channel Filter (Decorator Block)

To enhance artistic quality, we introduce a sub-network that predicts a $1 \times 1$ convolution filter based on both content and style. This allows for dynamic adjustment of feature channel interactions, resulting in sharper textures and superior color matching than standard methods.

Results and Discussion

The model was evaluated using the MPI Sintel dataset for quantitative temporal consistency and a diverse set of 20 artistic styles for zero-shot testing.

Quantitative Performance

On the MPI Sintel benchmark, the proposed method achieved the lowest temporal loss across all tested intervals. Notably, it outperformed established optimization-based methods as the frame interval increased, demonstrating superior long-term stability.

Stylized output frame demonstrating texture preservation and temporal stability.

Qualitative Analysis

The model successfully maintains complex artistic patterns, such as brush strokes, across frames even during significant object motion. Unlike post-processing filters, our approach avoids color casting and blurring issues. While a slight blurring of strokes is observed compared to non-regularized baselines—a necessary trade-off for stability—the overall artistic integrity remains high.

Conclusion

This paper demonstrates that mathematical regularization via $L_{comp}$ and global feature sharing can effectively replace or augment optical flow for video style transfer. The resulting feed-forward network is efficient enough for practical applications while providing state-of-the-art temporal consistency. Future work will explore reducing memory constraints for high-resolution sequences and investigating modern architectures like Transformers for inherent stability benefits.

Summary updated

BibTeX
@inproceedings{wang2020consistent,
  author = {Wenjing Wang and Jizheng Xu and Li Zhang and Yue Wang and Jiaying Liu},
  title = {{Consistent Video Style Transfer via Compound Regularization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6905},
}
Figure