F3-Pruning: A Training-Free and Generalized Pruning Strategy towards Faster and Finer Text-to-Video Synthesis

Sitong Su, Jianzhi Liu, Lianli Gao, Jingkuan Song

TL;DR

F3-Pruning is a training-free, generalized strategy that accelerates Text-to-Video (T2V) models by identifying and removing redundant temporal attention weights. By pruning these weights during the later stages of inference, the method achieves up to a 1.35x speedup while simultaneously improving video quality metrics like FVD by 22%.

What Problem Does This Paper Address?

Text-to-Video (T2V) synthesis has seen massive leaps in quality thanks to large-scale transformers and diffusion models. However, these models are computationally gluttonous. Generating even a short clip requires significant time and hardware resources.

Existing methods to speed up inference usually fall into two camps: they either require expensive retraining/fine-tuning or are "bespoke" solutions designed for one specific architecture. There is a distinct lack of a "plug-and-play" acceleration method that works across different model families (like transformers and diffusion models) without needing a single extra gradient step.

Methods

The authors introduce F3-Pruning (Faster and Finer), which targets the Temporal Attention (TA) modules common to most T2V architectures. Their approach is based on a deep dive into how these models actually process information during inference.

They categorize attention into three types:

  1. Cross-modal Attention (CA): Aligns text prompts with visual features.
  2. Self-Attention (SA): Ensures visual quality within a single frame.
  3. Temporal Attention (TA): Maintains coherence across different frames.
Overview of attention modules and the pruning schedule

The core innovation is the Aggregate Attention Score (AAS). Instead of complex, per-token comparisons, the authors simply sum the temporal attention values for each network layer or denoising timestep. They found that TA weights with low AAS are essentially "dead weight."

The F3-Pruning Pipeline: * Rank: Calculate AAS across the entire generation process (all layers/timesteps). * Prune: Physically remove the temporal attention weights whose AAS ranks in the bottom $\alpha$ proportion (the pruning ratio). * Redistribute: Because attention is normalized, removing TA naturally shifts the model's focus toward SA and CA in the later stages of generation.

Datasets

The authors evaluated their method on three distinct benchmarks:

  • UCF-101: A standard action recognition dataset used here for label-to-video generation.
  • DAVIS: A densely annotated video segmentation dataset.
  • LONGTEXT: A new dataset introduced by the authors consisting of complex, full-sentence prompts generated by ChatGPT to test the limits of text-visual alignment.

Key Insights and Ideas

  • Temporal Redundancy is Real: The authors observed that a huge portion of temporal attention values are near zero. Most frames only really "care" about their immediate neighbors, making dense connections across all frames unnecessary.
  • The "Shift of Importance": As the generation process progresses (deeper layers in transformers or later timesteps in diffusion), the importance of temporal information actually declines. The model shifts its focus from global structure to fine-grained visual details and text alignment.
  • Pruning as an Enhancer: Counter-intuitively, pruning TA doesn't just speed things up; it can make videos look better. By releasing the model from overly rigid temporal constraints in the late stages, the model can dedicate more "attention budget" to visual fidelity (SA) and following the text prompt (CA).
Figure 1: Demonstration of some visual results comparison in Text-to-Video (T2V) synthesis. Our F3-Pruning is applied to the classic transformer-based method CogVideo and the typical diffusion-based method Tune-A-Video. Without any extra training, F3-Pruning not only boosts inference efficiency of T2V but also enhances video quality. On the public video dataset UCF-101, applying F3-Pruning to CogVideo makes it 1.35x faster and promotes video quality metrics FVD by 22%.

Key Results

The researchers tested F3-Pruning on CogVideo (transformer-based) and Tune-A-Video (diffusion-based):

  • Inference Speed: Applying F3-Pruning to CogVideo resulted in a 1.35x speedup. Unlike other methods like ToMe, which can actually increase latency due to complex token-merging overhead, F3-Pruning provides a clean physical reduction in computation.
  • Video Quality: On the UCF-101 dataset, CogVideo with F3-Pruning saw a 22% improvement in Fréchet Video Distance (FVD), a key metric for video quality and coherence.
  • Text Alignment: Qualitative results on the LONGTEXT dataset showed that pruned models were often better at capturing complex actions described in the prompt compared to the baseline.

Limitations and Future Work

While F3-Pruning is highly effective, it isn't a "set it to 100%" solution. The authors found that a 50% pruning ratio is the "sweet spot." Pushing the ratio to 75% leads to a significant loss in inter-frame coherence, resulting in jittery or illogical videos.

Future work could involve dynamic pruning ratios that adapt to the complexity of the input text or exploring how this strategy interacts with newer, even larger models like Stable Video Diffusion or Sora-like architectures. Additionally, the authors primarily focused on the first stage of CogVideo (keyframe generation), leaving the impact on super-resolution stages for future exploration.

Summary updated

Abstract

Recently Text-to-Video (T2V) synthesis has undergone a breakthrough by training transformers or diffusion models on large-scale datasets. Nevertheless, inferring such large models incurs huge costs.Previous inference acceleration works either require costly retraining or are model-specific.To address this issue, instead of retraining we explore the inference process of two mainstream T2V models using transformers and diffusion models.The exploration reveals the redundancy in temporal attention modules of both models, which are commonly utilized to establish temporal relations among frames.Consequently, we propose a training-free and generalized pruning strategy called F3-Pruning to prune redundant temporal attention weights.Specifically, when aggregate temporal attention values are ranked below a certain ratio, corresponding weights will be pruned.Extensive experiments on three datasets using a classic transformer-based model CogVideo and a typical diffusion-based model Tune-A-Video verify the effectiveness of F3-Pruning in inference acceleration, quality assurance and broad applicability.
BibTeX
@inproceedings{su2024f,
  author = {Sitong Su and Jianzhi Liu and Lianli Gao and Jingkuan Song},
  title = {{F3-Pruning: A Training-Free and Generalized Pruning Strategy towards Faster and Finer Text-to-Video Synthesis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i5.28300},
}
Figure