Abstract
Style transfer has traditionally been divided into three distinct domains: artistic, photo-realistic, and video-based stylization. Achieving temporal consistency in video style transfer usually requires computationally expensive optical flow or rigid global constraints. This paper introduces the Contrastive Coherence Preserving Loss (CCPL), a novel approach that enforces local structure preservation through contrastive learning. When combined with a Simple Covariance Transformation (SCT) module, the resulting framework achieves high-quality, versatile stylization at real-time speeds—up to 77 FPS—without requiring video data during training.
Introduction
Style transfer—the process of applying the texture and color of a style image to the structure of a content image—is a cornerstone of neural art. However, the field has long been siloed. Artistic methods often fail to preserve photo-realistic details, and video stylization frequently suffers from "flicker," where independent frame processing leads to jarring temporal artifacts.
Previous attempts to solve video inconsistency relied heavily on optical flow to calculate pixel movement, which is both computationally intensive and prone to errors in occluded regions. Other "flow-free" methods often imposed rigid constraints that stifled the creative application of style. CCPL addresses these challenges by ensuring that the relationship between neighboring patches remains consistent after stylization, effectively stabilizing video output even when the model is trained solely on static images.
Methodology
Contrastive Coherence Preserving Loss (CCPL)
The core innovation of this work is the CCPL, which operates on the principle that if two patches are neighbors in the original content image, their stylized counterparts should maintain a similar spatial relationship. Rather than forcing stylized patches to be identical to the content (which would prevent stylization), CCPL utilizes Contrastive Learning (InfoNCE loss).
By maximizing the mutual information between the "difference" of patches in the content image and the "difference" of the same patches in the stylized image, CCPL acts as a neighbor-regulating mechanism. This suppresses local distortions and ensures that if a patch moves slightly across video frames, its stylization remains stable, thereby eliminating flicker without the need for explicit motion estimation.
Simple Covariance Transformation (SCT)
To fuse content and style features more effectively, the authors developed the SCT module. Traditional methods like AdaIN only align the mean and variance of features. SCT goes further by aligning second-order statistics (covariance). By capturing channel-wise correlations, SCT allows for more precise style capture. The architecture, known as SCTNet, utilizes a pre-trained VGG-19 encoder, the SCT fusion module, and a symmetric decoder to produce the final output.
Experimental Results
Performance and Speed
SCTNet is designed for efficiency, making it viable for real-time applications. It achieves a processing speed of 77.0 FPS at 256×256 resolution and 21.7 FPS at 512×512. This significantly outperforms previous state-of-the-art (SOTA) methods while maintaining high visual quality.
Temporal Consistency and Versatility
Quantitative evaluations on the MPI Sintel dataset show that CCPL-equipped models outperform existing flow-free methods and perform on par with complex multi-frame methods like ReReVST. Qualitatively, CCPL effectively removes "dirty" textures and checkerboard artifacts common in arbitrary style transfer.
Furthermore, the loss function is highly versatile. When integrated into the Contrastive Unpaired Translation (CUT) framework for image-to-image tasks (such as horse-to-zebra translation), it significantly reduced perceptual distance (LPIPS) and improved temporal stability in video translations.
Discussion and Limitations
While CCPL offers a robust solution for coherence, there is an inherent trade-off between stylization strength and temporal stability. Increasing the weight of the CCPL can lead to a slight decrease in the intensity of the applied style. Additionally, while SCT aligns statistics effectively, future work could improve semantic alignment to ensure that specific styles (e.g., "water" textures) are only applied to relevant content areas (e.g., a lake). Finally, while the method handles standard motion well, extremely fast or complex camera movements may still present challenges for perfect coherence without explicit motion compensation.
Conclusion
CCPL provides a powerful, flow-free mechanism for maintaining coherence across various style transfer tasks. By focusing on the "rhythm" of local patch differences rather than pixel-perfect identity, the method bridges the gap between image and video stylization, offering a versatile and high-performance solution for the next generation of neural art tools.