Introduction
Text-to-Video (T2V) synthesis has seen remarkable progress, yet the computational cost remains a significant barrier to widespread adoption. Modern T2V models, whether based on Transformers or Diffusion architectures, rely heavily on Temporal Attention (TA) to maintain frame-to-frame coherence. However, the complexity of TA grows quadratically with the number of frames, leading to massive memory and time requirements.
Existing acceleration techniques often require expensive retraining or are limited to specific model architectures. F³-Pruning introduces a "plug-and-play" strategy that is both training-free and generalized, achieving a 1.35x speedup while simultaneously improving video quality by redistributing attention focus.
The Challenge of Temporal Attention
In T2V models, attention is typically divided into three categories:
- Cross-modal Attention (CA): Aligns text prompts with visual content.
- Self-Attention (SA): Ensures visual quality and spatial consistency within individual frames.
- Temporal Attention (TA): Manages the logical flow and motion between frames.
The researchers identified a critical "temporal redundancy" in these models. While TA is essential during the early stages of generation to establish motion, its importance diminishes significantly as the process nears completion. In later layers or denoising timesteps, TA weights often approach zero, becoming a computational burden that offers little to no contribution to the final output.
The F³-Pruning Mechanism
F³-Pruning leverages this redundancy through a simple yet effective three-step process:
- Aggregate Attention Score (AAS): The model calculates a score for each layer or timestep by summing its temporal attention values.
- Ranking and Pruning: These AAS values are ranked. If a layer's score falls below a predefined pruning ratio ($\alpha$), its temporal weights are pruned.
- Attention Redistribution: Because attention values are normalized to sum to 1, pruning the redundant TA weights naturally shifts the model's focus toward CA and SA.
This redistribution is the key to the method's success. By "releasing" the model from temporal constraints in the final stages, it can dedicate more resources to high-frequency visual details and tighter alignment with the text prompt.
Experimental Results
The authors evaluated F³-Pruning across multiple benchmarks, including UCF-101, DAVIS, and a new complex prompt benchmark called LONGTEXT.
Speed and Efficiency
On the CogVideo model, F³-Pruning achieved a practical 1.35x speedup and reduced floating-point operations (FLOPs) by 44%. Unlike other methods like Token Merging (ToMe), which can introduce overhead that negates speed gains, F³-Pruning's structural approach ensures tangible real-world performance improvements.
Quality Improvement
Counter-intuitively, pruning TA actually improves video quality. On the UCF-101 dataset, the Frechet Video Distance (FVD)—a standard metric for video quality—improved by 22%. Qualitative results demonstrate that F³-Pruning avoids the static or illogical motion sometimes seen in baseline models, producing sharper details and more fluid movements.
Conclusion
F³-Pruning represents a significant step forward in making high-quality video generation more accessible. By identifying and removing temporal redundancy without requiring retraining, it offers a universal solution for accelerating both Transformer and Diffusion-based T2V models. Future work may explore dynamic pruning ratios to automatically adapt to varying motion complexities in user prompts, further optimizing the balance between speed and coherence.
Summary updated