TITAN-Guide: Taming Inference-Time Alignment for Guided Text-to-Video Diffusion Models

Christian Simon, Masato Ishii, Akio Hayakawa, Zhi Zhong, Shusuke Takahashi, Takashi Shibuya, Yuki Mitsufuji

TL;DR

This paper introduces TITAN-Guide, a novel method for efficiently guiding Text-to-Video (T2V) diffusion models at inference time. By leveraging forward gradient descents and novel gradient guessing strategies, TITAN-Guide significantly reduces memory consumption, enabling high-resolution video generation on consumer hardware, while achieving state-of-the-art performance across various guidance tasks.

What Problem Does This Paper Address?

Conditional diffusion models, particularly for Text-to-Video (T2V) generation, have shown remarkable capabilities. However, achieving fine-grained control over generated content (e.g., specific styles, audio alignment, or aesthetic quality) often requires extensive supervised fine-tuning of the base model, which is computationally expensive and time-consuming. While "training-free" guidance methods offer a more efficient alternative by using off-the-shelf models to steer generation, existing approaches suffer from high memory requirements, especially for memory-intensive T2V models, and can lead to sub-optimal control due to rough gradient estimations. This gap limits the practical application of guided diffusion for high-resolution, multi-frame video generation on standard hardware.

Methods

TITAN-Guide tackles these challenges by introducing a training-free guidance framework that operates efficiently at inference time. The core innovation lies in its use of forward gradient descents instead of traditional backpropagation for computing gradients. This approach significantly reduces memory overhead.

Key components of TITAN-Guide include:

  • Forward Gradient Descents: This technique allows for gradient computation without the need for a backward pass, drastically cutting down on memory usage. It leverages a "gradient guess" vector to project the gradient direction.
  • Novel Gradient Guessing Strategies: The paper explores and proposes three strategies for generating the "gradient guess" vector:
  • Random Guesses: Using a standard Gaussian distribution.
  • Score-based Guesses: Leveraging the output of the denoiser itself to provide a more informed, deterministic guess.
  • Sampled Gradient Guesses: Estimating gradients using a subset of frames to further reduce computational cost.
  • Latent Space Manipulation: The method operates in the latent space of diffusion models, which is more memory-efficient than working directly with pixel data.
  • Efficient Decoder Usage: TITAN-Guide minimizes the use of the decoder, which is a major memory bottleneck in T2V models, by strategically applying guidance.

Datasets

The paper evaluates TITAN-Guide on the following datasets:

  • VGG-Sound [4]: Used for audio-video alignment tasks. The authors constructed a custom evaluation dataset by randomly selecting 3,000 video-audio pairs.
  • TFG-1000 prompts [31]: Used for aesthetic-score and style guidance tasks.
  • Moons dataset: Used for a toy experiment to simulate classifier guidance and evaluate different gradient guess strategies.

Key Insights and Ideas

  • Forward Gradients are Key for Memory Efficiency: The paper convincingly demonstrates that moving away from backpropagation to forward gradients is crucial for enabling guided diffusion on memory-constrained hardware, especially for T2V.
  • Gradient Guessing Matters: The choice of how to estimate the gradient direction significantly impacts performance. Score-based and sampled gradient guesses offer a better trade-off between accuracy and efficiency than purely random guesses.
  • Latent Space is the Way to Go: Operating in the compressed latent space of T2V models is essential for managing the computational and memory demands of video generation.
  • Early Iterations are Sensitive: The paper highlights that steering diffusion latents in later iterations has less impact than in earlier ones, as the video's appearance has already taken shape. This informs the strategy of focusing guidance where it's most effective.

Key Results

  • Significant Memory Reduction: TITAN-Guide uses substantially less memory than existing methods. For example, at 256x256 resolution, it uses around 20.6GB, while baselines like DOODL can exceed 67.2GB. At higher resolutions (384x384), TITAN-Guide uses ~42GB, whereas baselines struggle to fit on high-end GPUs.
  • State-of-the-Art Performance: TITAN-Guide achieves competitive or superior results across various guidance tasks, including audio-video alignment (lowest FVD of 168.88 for frame interpolation, improved AV-Align scores), aesthetic guidance (highest DOVER scores), and style guidance.
  • Effective Gradient Strategies: Score-based and sampled gradient guesses in TITAN-Guide closely approximate ground truth gradients in toy experiments, demonstrating their effectiveness.
  • High-Resolution Generation: The method enables high-resolution video generation (up to 384x384) on consumer GPUs, a feat difficult for previous methods.

Limitations and Future Work

  • Sub-optimal Control with Rough Estimation: While TITAN-Guide improves upon existing methods, the paper acknowledges that rough estimation can still limit applicability in certain scenarios.
  • Future Work: The authors suggest exploring more advanced gradient approximation techniques and further optimizing the guidance process for even greater efficiency and control. The paper also implicitly points towards the potential for TITAN-Guide to be applied to other memory-intensive generative tasks beyond T2V.

Summary updated

Abstract

In the recent development of conditional diffusion models still require heavy supervised fine-tuning for performing control on a category of tasks. Training-free conditioning via guidance with off-the-shelf models is a favorable alternative to avoid further fine-tuning on the base model. However, the existing training-free guidance frameworks either have heavy memory requirements or offer sub-optimal control due to rough estimation. These shortcomings limit the applicability to control diffusion models that require intense computation, such as Text-to-Video (T2V) diffusion models. In this work, we propose Taming Inference Time Alignment for Guided Text-to-Video Diffusion Model, so-called TITAN-Guide, which overcomes memory space issues, and provides more optimal control in the guidance process compared to the counterparts. In particular, we develop an efficient method for optimizing diffusion latents without backpropagation from a discriminative guiding model. In particular, we study forward gradient descents for guided diffusion tasks with various options on directional directives. In our experiments, we demonstrate the effectiveness of our approach in efficiently managing memory during latent optimization, while previous methods fall short. Our proposed approach not only minimizes memory requirements but also significantly enhances T2V performance across a range of diffusion guidance benchmarks. Code, models, and demo are available at https://titanguide.github.io.
BibTeX
@inproceedings{simon2025titan,
  author = {Christian Simon and Masato Ishii and Akio Hayakawa and Zhi Zhong and Shusuke Takahashi and Takashi Shibuya and Yuki Mitsufuji},
  title = {{TITAN-Guide: Taming Inference-Time Alignment for Guided Text-to-Video Diffusion Models}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure