FancyVideo: Towards Dynamic and Consistent Video Generation via Cross-frame Textual Guidance

Jiasong Feng, Ao Ma, Jing Wang, Ke Cao, Zhanjie Zhang

Introduction

Text-to-video (T2V) generation has seen rapid progress, yet many state-of-the-art models suffer from a "static video" curse. While these models produce high-quality individual frames, the resulting videos often lack significant motion or temporal logic. FancyVideo addresses this by introducing a Cross-frame Textual Guidance Module (CTGM), moving beyond the traditional "one-prompt-fits-all-frames" approach to provide frame-specific textual control.

The Problem: Static Guidance for Dynamic Content

Most current T2V models are built by extending text-to-image (T2I) architectures. These models typically employ spatial cross-attention, which applies the exact same text prompt embedding to every frame in a sequence. This creates a fundamental flaw: if the guidance is identical for every frame, the model struggles to understand how a "verb" (like running, floating, or rising) should evolve over time.

Visualizing the attention maps of older models reveals that the regions focused on verbs remain nearly identical across the entire timeline. This lack of temporal evolution in textual guidance results in videos that are either nearly static or exhibit illogical, flickering motion.

Methods: The Cross-frame Textual Guidance Module (CTGM)

The core innovation of FancyVideo is the CTGM, which replaces standard spatial cross-attention with a dynamic, time-aware mechanism. As shown in the architectural comparison below, CTGM is integrated into the spatial blocks of a pseudo-3D UNet and consists of three primary components:

The CTGM architecture (right) compared to standard spatial cross-attention (left), highlighting the injection of temporal information into the text-to-video pipeline.
  1. Temporal Information Injector (TII): Instead of a static embedding, TII processes the noisy latent features of the video using temporal self-attention. It then "injects" this frame-specific context into the text embeddings, creating unique textual conditions for each moment in time.
  2. Temporal Affinity Refiner (TAR): This module refines the correlation (affinity) matrix between latent video features and text. It ensures that the relationship between specific words (like a verb) and specific spatial patches evolves logically along the time dimension.
  3. Temporal Feature Booster (TFB): A final refinement step at the end of the cross-attention process that further enhances temporal plausibility and motion consistency.

Additionally, FancyVideo employs a Motion Embedding strategy. By calculating motion scores using optical flow (RAFT) during training, the model allows users to specify a desired "motion score" during inference, providing explicit control over the intensity of movement.

Key Insights: Visualizing the "Verb" Focus

The effectiveness of CTGM is most apparent when visualizing how the model attends to specific verbs in a prompt. In baseline models like AnimateDiff, the attention maps for a verb remain static. In contrast, FancyVideo's attention maps shift logically as the action progresses, allowing the model to map verbs to actual, consistent movement.

Comparison of synthesized videos and verb-focused attention maps. FancyVideo shows noticeable changes in attention over time, corresponding to more dynamic and logical motion.

Results and Performance

FancyVideo was evaluated against industry-standard benchmarks including EvalCrafter, UCF-101, and MSR-VTT. Key findings include:

  • State-of-the-Art Quality: FancyVideo achieved the highest scores for Video Quality (177.72) and Text-Video Alignment (396.65) on EvalCrafter, outperforming established models like Gen-2 and Pika.
  • Superior Motion Balance: While some models produce high raw motion at the cost of visual fidelity, FancyVideo maintains high aesthetic quality while achieving significantly better motion scores than most competitors.
  • Human Preference: In blind trials, human evaluators significantly preferred FancyVideo across all categories, including video quality, text alignment, motion quality, and temporal consistency.

Conclusion and Future Work

By making text embeddings "aware" of the specific frames they guide, FancyVideo successfully bridges the gap between static image quality and dynamic video logic. While the model currently faces challenges such as occasional watermark persistence inherited from its T2I foundations (SDXL), it represents a significant step toward more controllable and temporally consistent AI video generation. Future work will focus on optimizing computational overhead and extending these techniques to long-form video generation.

Summary updated

Abstract

Synthesizing motion-rich and temporally consistent videos remains a challenge in artificial intelligence, especially when dealing with extended durations. Existing text-to-video (T2V) models commonly employ spatial cross-attention for text control, equivalently guiding different frame generations without frame-specific textual guidance. Thus, the model's capacity to comprehend the temporal logic conveyed in prompts and generate videos with coherent motion is restricted. To tackle this limitation, we introduce FancyVideo, an innovative video generator that improves the existing text-control mechanism with the well-designed Cross-frame Textual Guidance Module (CTGM). Specifically, CTGM incorporates the Temporal Information Injector (TII) and Temporal Affinity Refiner (TAR) at the beginning and end of cross-attention, respectively, to achieve frame-specific textual guidance. Firstly, TII injects frame-specific information from latent features into text conditions, thereby obtaining cross-frame textual conditions. Then, TAR refines the correlation matrix between cross-frame textual conditions and latent features along the time dimension. Extensive experiments comprising both quantitative and qualitative evaluations demonstrate the effectiveness of FancyVideo. Our approach achieves state-of-the-art T2V generation results on the EvalCrafter benchmark and facilitates the synthesis of dynamic and consistent videos. Note that the T2V process of FancyVideo essentially involves a text-to-image step followed by T+I2V. This means it also supports the generation of videos from user images, i.e., the image-to-video (I2V) task. A significant number of experiments have shown that its performance is also outstanding.
BibTeX
@inproceedings{feng2025fancyvideo,
  author = {Jiasong Feng and Ao Ma and Jing Wang and Ke Cao and Zhanjie Zhang},
  title = {{FancyVideo: Towards Dynamic and Consistent Video Generation via Cross-frame Textual Guidance}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/1120},
}
Figure