Video-T1: Test-time Scaling for Video Generation

TL;DR

This paper introduces Video-T1, a novel framework for improving video generation quality by scaling computation at inference time rather than during training. By reframing video generation as a search problem, Video-T1 and its efficient Tree-of-Frames (ToF) algorithm significantly enhance video quality and text alignment with minimal additional training cost.

What Problem Does This Paper Address?

The field of video generation has seen remarkable progress, largely driven by scaling up models and training data. However, this approach is computationally expensive and resource-intensive. While researchers have found success in applying "Test-Time Scaling" (TTS) to Large Language Models (LLMs) to improve performance without retraining, this concept has been less explored for video generation. The core challenge is that video generation requires maintaining temporal coherence across frames, making it more complex than text generation. This paper aims to answer: "If we allow video generation models to use more inference-time computation, how much can we improve generation quality for challenging text prompts?"

Methods

The paper proposes Video-T1, a framework that reinterprets test-time scaling for video generation as a search problem. The goal is to find better video "trajectories" within the Gaussian noise space that lead to the target video distribution. This involves:

• Video Generator (G): The standard text-to-video diffusion model.
• Test Verifiers (V): Multimodal evaluation models that assess the quality of generated videos and provide feedback. These can be off-the-shelf models like VisionReward, VideoScore, and VideoLLaMA3.
• Heuristic Search Algorithms (f): Methods that use verifier feedback to guide the search for optimal video sequences.

Two search strategies are introduced:

1. Random Linear Search: A baseline approach where multiple initial noise samples are generated, each producing a full video. The best video is selected based on verifier scores. This scales linearly with the number of samples but can be computationally intensive.
2. Tree-of-Frames (ToF) Search: A more efficient, autoregressive approach that models video generation as a tree structure. It adaptively expands and prunes video "branches" (sequences of frames) based on verifier feedback. ToF incorporates:
3. Image-level alignment: Dynamically scales denoising computation by evaluating intermediate frames for clarity and potential.
4. Hierarchical prompting: Uses prompts at different stages (first frame, intermediate frames, final frames) to guide verifiers for consistency, motion, and overall alignment.
5. Heuristic pruning: Selects the most promising video paths to explore, reducing computational cost.

Datasets

The paper evaluates its methods on existing text-conditioned video generation benchmarks. Specific datasets are not detailed, but the evaluation uses models trained on large-scale video datasets.

Key Insights and Ideas

• Test-Time Scaling is Viable for Video: The core insight is that simply allocating more computational budget during inference can significantly improve video generation quality, similar to LLMs, without requiring expensive retraining.
• Video Generation as a Search Problem: Reframing the task as searching for optimal video trajectories in a high-dimensional space is key to leveraging TTS effectively.
• Tree-of-Frames (ToF) Efficiency: The ToF algorithm is a crucial innovation, offering a more computationally efficient way to explore the search space compared to random linear search, especially for longer videos. Its adaptive pruning and branching are critical for balancing quality and cost.
• Verifier Importance: The choice and combination of verifiers significantly impact the effectiveness of TTS. Using multiple verifiers can mitigate biases and lead to better results.
• Model Scale Matters: Larger foundation models benefit more from TTS, suggesting that their inherent capacity can be better exploited with increased inference computation.

Key Results

• Consistent Improvements: TTS consistently leads to significant improvements in video quality and text-video alignment across various video generation models.
• ToF Efficiency: Figure 4 shows that ToF search achieves comparable or better performance than random linear search at a much lower computational cost. For example, for Pyramid-Flow (SD3), ToF achieves a similar score with significantly fewer GFLOPs.
• Quantitative Gains: Table 1 shows substantial percentage improvements in various VBench metrics (e.g., Total Score, Quality Score, Semantic Score) for models with TTS. For CogVideoX-5B, the "Multiple Objects" score improved by 36.0% with TTS.
• Prompt Complexity: Figure 5 illustrates that TTS provides improvements across different prompt complexities, though some dimensions like "Motion Smoothness" and "Temporal Flickering" are more challenging to improve due to the inherent difficulty in precisely controlling motion and consistency.

Limitations and Future Work

• Challenging Dimensions: The paper notes that certain video properties, like precise motion smoothness and temporal flickering, are harder to improve with current TTS methods and verifiers. This is attributed to the difficulty in precisely controlling motion trajectories and the limitations of current multimodal verifiers in assessing fine-grained temporal consistency.
• Verifier Limitations: While multiple verifiers help, they still have limitations in perfectly assessing all aspects of video quality. Future work could focus on developing more sophisticated verifiers.
• Future Directions: The paper suggests that this framework opens new avenues for research into efficient test-time optimization strategies in video generation, particularly for leveraging advanced foundation models. Further exploration of adaptive branching and pruning strategies within ToF could also yield further gains.