ConCLVD: Controllable Chinese Landscape Video Generation via Diffusion Model

Dingming Liu, Shaowei Li, Ruoyan Zhou, Lili Liang, Yongguan Hong, Fei Chao, Rongrong JiarXiv 2024 · DOI · OpenReview

Introduction

Text-to-video (T2V) diffusion models have achieved remarkable success in generating realistic and diverse content. However, these general-purpose models often struggle to capture the nuanced aesthetic of niche artistic styles, such as traditional Chinese landscape painting (shanshui). This art form relies on specific techniques like "ink wash," the "five shades of ink," and a unique spatial dynamism that generic models like Pika 1.0 or Gen-2 frequently fail to replicate, often introducing modern artifacts that clash with the traditional style.

Comparison of ConCLVD with existing T2V models, highlighting the stylistic fidelity and coherence achieved through the CLV-HD dataset.

The authors of ConCLVD identify three primary challenges in this domain: the scarcity of high-quality specialized datasets, the difficulty of maintaining both local and global temporal coherence, and the prohibitive computational costs associated with training state-of-the-art T2V models. ConCLVD addresses these by introducing a specialized dataset, a novel dual-attention motion module, and an efficient training strategy that runs on a single consumer-grade GPU.

The ConCLVD Framework

ConCLVD is built upon a frozen Stable Diffusion v1.5 backbone, augmented with three key technical innovations designed to enhance temporal consistency and stylistic authenticity.

Dual-Attention Motion Module

To balance global consistency with local fluid motion, the framework employs a specialized motion module. It utilizes Versatile Attention, which considers the entire video sequence to maintain global patterns and the overall "vibe" of the landscape, and Sparse-Causal Attention, which focuses on the immediate preceding frame to ensure smooth, logical transitions for local elements like moving birds or swaying bamboo.

Unsupervised Contrastive Learning (Noise Adapter)

To improve the model's understanding of temporal dynamics without requiring extensive manual labeling, the researchers introduced a Noise Adapter. This component applies contrastive learning directly to the predicted noise in the latent space. By treating adjacent frames as positive pairs and distant frames as negative pairs, the model learns a more robust representation of the "rhythm" and distance between frames.

The ConCLVD training and inference pipeline, illustrating the integration of the Noise Adapter, Dual-Attention modules, and SOFP interpolation.

Sparse Optical Flow Projection (SOFP)

For the inference stage, the authors propose SOFP Interpolation, a plug-and-play method that uses optical flow to project and fill intermediate frames. This significantly increases video smoothness during generation without requiring additional training parameters.

Dataset: CLV-HD

A core contribution of this work is the CLV-HD (Chinese Landscape Video-High Definition) dataset. It consists of 1,300 high-definition text-video pairs, curated from Chinese water ink animations and high-quality landscape footage. Each clip was manually segmented and annotated with detailed text descriptions to ensure precise alignment between stylistic prompts and visual output.

Key Results and Discussion

ConCLVD demonstrates superior performance in both quantitative and qualitative evaluations. In terms of temporal consistency, ConCLVD achieved a CLIP-temp score of 0.983, outperforming established baselines such as Gen-2 (0.978) and AnimateLCM (0.970).

Qualitative user studies involving 100 participants confirmed these findings, with ConCLVD being the preferred model for "Adherence to Chinese landscape painting style" and "Overall Quality." Ablation studies further highlighted the importance of the individual components: removing Sparse-Causal Attention resulted in "jumpy" transitions, while omitting the contrastive Noise Adapter led to static videos lacking dynamic movement.

Conclusion

ConCLVD successfully bridges the gap between modern generative AI and traditional artistic expression. By prioritizing stylistic fidelity and computational efficiency, the framework enables high-quality artistic video generation on accessible hardware (e.g., a single NVIDIA RTX 3090). Future work will focus on improving text-semantic alignment and exploring the application of this controllable framework to other traditional art forms.

Summary updated

Abstract

Chinese landscape painting is a gem of Chinese cultural and artistic heritage that showcases the splendor of nature through the deep observations and imaginations of its painters. Limited by traditional techniques, these artworks were confined to static imagery in ancient times, leaving the dynamism of landscapes and the subtleties of artistic sentiment to the viewer's imagination. Recently, emerging text-to-video (T2V) diffusion methods have shown significant promise in video generation, providing hope for the creation of dynamic Chinese landscape paintings. However, challenges such as the lack of specific datasets, the intricacy of artistic styles, and the creation of extensive, high-quality videos pose difficulties for these models in generating Chinese landscape painting videos. In this paper, we propose CLV-HD (Chinese Landscape Video-High Definition), a novel T2V dataset for Chinese landscape painting videos, and ConCLVD (Controllable Chinese Landscape Video Diffusion), a T2V model that utilizes Stable Diffusion. Specifically, we present a motion module featuring a dual attention mechanism to capture the dynamic transformations of landscape imageries, alongside a noise adapter to leverage unsupervised contrastive learning in the latent space. Following the generation of keyframes, we employ optical flow for frame interpolation to enhance video smoothness. Our method not only retains the essence of the landscape painting imageries but also achieves dynamic transitions, significantly advancing the field of artistic video generation. The source code and dataset are available at https://anonymous.4open.science/r/ConCLVD-EFE3.
BibTeX
@misc{liu2024conclvd,
  author = {Dingming Liu and Shaowei Li and Ruoyan Zhou and Lili Liang and Yongguan Hong and Fei Chao and Rongrong Ji},
  title = {{ConCLVD: Controllable Chinese Landscape Video Generation via Diffusion Model}},
  year = {2024},
  eprint = {2404.12903},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.12903},
}
Figure