Introduction
Text-to-video (T2V) diffusion models have achieved remarkable success in generating realistic and diverse content. However, these general-purpose models often struggle to capture the nuanced aesthetic of niche artistic styles, such as traditional Chinese landscape painting (shanshui). This art form relies on specific techniques like "ink wash," the "five shades of ink," and a unique spatial dynamism that generic models like Pika 1.0 or Gen-2 frequently fail to replicate, often introducing modern artifacts that clash with the traditional style.
The authors of ConCLVD identify three primary challenges in this domain: the scarcity of high-quality specialized datasets, the difficulty of maintaining both local and global temporal coherence, and the prohibitive computational costs associated with training state-of-the-art T2V models. ConCLVD addresses these by introducing a specialized dataset, a novel dual-attention motion module, and an efficient training strategy that runs on a single consumer-grade GPU.
The ConCLVD Framework
ConCLVD is built upon a frozen Stable Diffusion v1.5 backbone, augmented with three key technical innovations designed to enhance temporal consistency and stylistic authenticity.
Dual-Attention Motion Module
To balance global consistency with local fluid motion, the framework employs a specialized motion module. It utilizes Versatile Attention, which considers the entire video sequence to maintain global patterns and the overall "vibe" of the landscape, and Sparse-Causal Attention, which focuses on the immediate preceding frame to ensure smooth, logical transitions for local elements like moving birds or swaying bamboo.
Unsupervised Contrastive Learning (Noise Adapter)
To improve the model's understanding of temporal dynamics without requiring extensive manual labeling, the researchers introduced a Noise Adapter. This component applies contrastive learning directly to the predicted noise in the latent space. By treating adjacent frames as positive pairs and distant frames as negative pairs, the model learns a more robust representation of the "rhythm" and distance between frames.
Sparse Optical Flow Projection (SOFP)
For the inference stage, the authors propose SOFP Interpolation, a plug-and-play method that uses optical flow to project and fill intermediate frames. This significantly increases video smoothness during generation without requiring additional training parameters.
Dataset: CLV-HD
A core contribution of this work is the CLV-HD (Chinese Landscape Video-High Definition) dataset. It consists of 1,300 high-definition text-video pairs, curated from Chinese water ink animations and high-quality landscape footage. Each clip was manually segmented and annotated with detailed text descriptions to ensure precise alignment between stylistic prompts and visual output.
Key Results and Discussion
ConCLVD demonstrates superior performance in both quantitative and qualitative evaluations. In terms of temporal consistency, ConCLVD achieved a CLIP-temp score of 0.983, outperforming established baselines such as Gen-2 (0.978) and AnimateLCM (0.970).
Qualitative user studies involving 100 participants confirmed these findings, with ConCLVD being the preferred model for "Adherence to Chinese landscape painting style" and "Overall Quality." Ablation studies further highlighted the importance of the individual components: removing Sparse-Causal Attention resulted in "jumpy" transitions, while omitting the contrastive Noise Adapter led to static videos lacking dynamic movement.
Conclusion
ConCLVD successfully bridges the gap between modern generative AI and traditional artistic expression. By prioritizing stylistic fidelity and computational efficiency, the framework enables high-quality artistic video generation on accessible hardware (e.g., a single NVIDIA RTX 3090). Future work will focus on improving text-semantic alignment and exploring the application of this controllable framework to other traditional art forms.
Summary updated