F³-Pruning: A Training-Free and Generalized Pruning Strategy towards Faster and Finer Text-to-Video Synthesis

Sitong Su, Jianzhi Liu, Lianli Gao, Jingkuan Song

Introduction

Text-to-Video (T2V) synthesis has seen remarkable progress, yet the computational cost remains a significant barrier to widespread adoption. Modern T2V models, whether based on Transformers or Diffusion architectures, rely heavily on Temporal Attention (TA) to maintain frame-to-frame coherence. However, the complexity of TA grows quadratically with the number of frames, leading to massive memory and time requirements.

Existing acceleration techniques often require expensive retraining or are limited to specific model architectures. F³-Pruning introduces a "plug-and-play" strategy that is both training-free and generalized, achieving a 1.35x speedup while simultaneously improving video quality by redistributing attention focus.

The Challenge of Temporal Attention

In T2V models, attention is typically divided into three categories:

  • Cross-modal Attention (CA): Aligns text prompts with visual content.
  • Self-Attention (SA): Ensures visual quality and spatial consistency within individual frames.
  • Temporal Attention (TA): Manages the logical flow and motion between frames.

The researchers identified a critical "temporal redundancy" in these models. While TA is essential during the early stages of generation to establish motion, its importance diminishes significantly as the process nears completion. In later layers or denoising timesteps, TA weights often approach zero, becoming a computational burden that offers little to no contribution to the final output.

The F³-Pruning Mechanism

F³-Pruning leverages this redundancy through a simple yet effective three-step process:

  1. Aggregate Attention Score (AAS): The model calculates a score for each layer or timestep by summing its temporal attention values.
  2. Ranking and Pruning: These AAS values are ranked. If a layer's score falls below a predefined pruning ratio ($\alpha$), its temporal weights are pruned.
  3. Attention Redistribution: Because attention values are normalized to sum to 1, pruning the redundant TA weights naturally shifts the model's focus toward CA and SA.
Overview of the F³-Pruning strategy across Transformer and Diffusion-based models.

This redistribution is the key to the method's success. By "releasing" the model from temporal constraints in the final stages, it can dedicate more resources to high-frequency visual details and tighter alignment with the text prompt.

Experimental Results

The authors evaluated F³-Pruning across multiple benchmarks, including UCF-101, DAVIS, and a new complex prompt benchmark called LONGTEXT.

Speed and Efficiency

On the CogVideo model, F³-Pruning achieved a practical 1.35x speedup and reduced floating-point operations (FLOPs) by 44%. Unlike other methods like Token Merging (ToMe), which can introduce overhead that negates speed gains, F³-Pruning's structural approach ensures tangible real-world performance improvements.

Quality Improvement

Counter-intuitively, pruning TA actually improves video quality. On the UCF-101 dataset, the Frechet Video Distance (FVD)—a standard metric for video quality—improved by 22%. Qualitative results demonstrate that F³-Pruning avoids the static or illogical motion sometimes seen in baseline models, producing sharper details and more fluid movements.

Qualitative comparison showing improved visual fidelity and motion with F³-Pruning.

Conclusion

F³-Pruning represents a significant step forward in making high-quality video generation more accessible. By identifying and removing temporal redundancy without requiring retraining, it offers a universal solution for accelerating both Transformer and Diffusion-based T2V models. Future work may explore dynamic pruning ratios to automatically adapt to varying motion complexities in user prompts, further optimizing the balance between speed and coherence.

Summary updated

Abstract

Recently Text-to-Video (T2V) synthesis has undergone a breakthrough by training transformers or diffusion models on large-scale datasets. Nevertheless, inferring such large models incurs huge costs.Previous inference acceleration works either require costly retraining or are model-specific.To address this issue, instead of retraining we explore the inference process of two mainstream T2V models using transformers and diffusion models.The exploration reveals the redundancy in temporal attention modules of both models, which are commonly utilized to establish temporal relations among frames.Consequently, we propose a training-free and generalized pruning strategy called F3-Pruning to prune redundant temporal attention weights.Specifically, when aggregate temporal attention values are ranked below a certain ratio, corresponding weights will be pruned.Extensive experiments on three datasets using a classic transformer-based model CogVideo and a typical diffusion-based model Tune-A-Video verify the effectiveness of F3-Pruning in inference acceleration, quality assurance and broad applicability.
BibTeX
@inproceedings{su2024f,
  author = {Sitong Su and Jianzhi Liu and Lianli Gao and Jingkuan Song},
  title = {{F³-Pruning: A Training-Free and Generalized Pruning Strategy towards Faster and Finer Text-to-Video Synthesis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i5.28300},
}
Figure