Introduction
Text-to-video (T2V) generation has seen rapid progress, yet many state-of-the-art models suffer from a "static video" curse. While these models produce high-quality individual frames, the resulting videos often lack significant motion or temporal logic. FancyVideo addresses this by introducing a Cross-frame Textual Guidance Module (CTGM), moving beyond the traditional "one-prompt-fits-all-frames" approach to provide frame-specific textual control.
The Problem: Static Guidance for Dynamic Content
Most current T2V models are built by extending text-to-image (T2I) architectures. These models typically employ spatial cross-attention, which applies the exact same text prompt embedding to every frame in a sequence. This creates a fundamental flaw: if the guidance is identical for every frame, the model struggles to understand how a "verb" (like running, floating, or rising) should evolve over time.
Visualizing the attention maps of older models reveals that the regions focused on verbs remain nearly identical across the entire timeline. This lack of temporal evolution in textual guidance results in videos that are either nearly static or exhibit illogical, flickering motion.
Methods: The Cross-frame Textual Guidance Module (CTGM)
The core innovation of FancyVideo is the CTGM, which replaces standard spatial cross-attention with a dynamic, time-aware mechanism. As shown in the architectural comparison below, CTGM is integrated into the spatial blocks of a pseudo-3D UNet and consists of three primary components:
- Temporal Information Injector (TII): Instead of a static embedding, TII processes the noisy latent features of the video using temporal self-attention. It then "injects" this frame-specific context into the text embeddings, creating unique textual conditions for each moment in time.
- Temporal Affinity Refiner (TAR): This module refines the correlation (affinity) matrix between latent video features and text. It ensures that the relationship between specific words (like a verb) and specific spatial patches evolves logically along the time dimension.
- Temporal Feature Booster (TFB): A final refinement step at the end of the cross-attention process that further enhances temporal plausibility and motion consistency.
Additionally, FancyVideo employs a Motion Embedding strategy. By calculating motion scores using optical flow (RAFT) during training, the model allows users to specify a desired "motion score" during inference, providing explicit control over the intensity of movement.
Key Insights: Visualizing the "Verb" Focus
The effectiveness of CTGM is most apparent when visualizing how the model attends to specific verbs in a prompt. In baseline models like AnimateDiff, the attention maps for a verb remain static. In contrast, FancyVideo's attention maps shift logically as the action progresses, allowing the model to map verbs to actual, consistent movement.
Results and Performance
FancyVideo was evaluated against industry-standard benchmarks including EvalCrafter, UCF-101, and MSR-VTT. Key findings include:
- State-of-the-Art Quality: FancyVideo achieved the highest scores for Video Quality (177.72) and Text-Video Alignment (396.65) on EvalCrafter, outperforming established models like Gen-2 and Pika.
- Superior Motion Balance: While some models produce high raw motion at the cost of visual fidelity, FancyVideo maintains high aesthetic quality while achieving significantly better motion scores than most competitors.
- Human Preference: In blind trials, human evaluators significantly preferred FancyVideo across all categories, including video quality, text alignment, motion quality, and temporal consistency.
Conclusion and Future Work
By making text embeddings "aware" of the specific frames they guide, FancyVideo successfully bridges the gap between static image quality and dynamic video logic. While the model currently faces challenges such as occasional watermark persistence inherited from its T2I foundations (SDXL), it represents a significant step toward more controllable and temporally consistent AI video generation. Future work will focus on optimizing computational overhead and extending these techniques to long-form video generation.
Summary updated