TITAN-Guide: Taming Inference-Time AligNment for Guided Text-to-Video Diffusion Models

Christian Simon, Masato Ishii, Akio Hayakawa, Zhi Zhong, Shusuke Takahashi, Takashi Shibuya, Yuki Mitsufuji

Introduction

Conditional diffusion models have revolutionized content generation, yet precisely controlling their output often necessitates resource-heavy supervised fine-tuning, such as training a ControlNet. Training-free guidance offers a more flexible alternative by using off-the-shelf classifiers (e.g., CLIP, ImageBind) to steer the diffusion process. However, applying this to Text-to-Video (T2V) models introduces two major hurdles: the "Estimation Problem," where early-stage noise leads to poor guidance signals, and the "Memory Problem," where backpropagating through video decoders requires VRAM far exceeding the capacity of consumer hardware.

Methodology: Forward Gradient Descent

To overcome these bottlenecks, the authors introduce TITAN-Guide (Taming Inference-Time AligNment). The core innovation lies in replacing traditional backpropagation with forward gradient descent.

While standard methods rely on Tweedie’s formula for a rough estimate of the clean video, TITAN-Guide samples the latent all the way to $t=0$. To avoid the massive memory overhead of storing activation graphs for backpropagation through the unrolled process and video decoder, TITAN-Guide utilizes forward-mode automatic differentiation. This computes directional derivatives in a single forward pass.

The researchers explored three strategies for generating the necessary "gradient guesses":

  1. Random guesses: Unbiased but high-variance isotropic Gaussians.
  2. Score-based guesses: Utilizing the denoiser's deterministic output.
  3. Sampled gradient guesses: Computing exact backpropagated gradients on a small subset (e.g., 2 frames) and extending that vector to the entire sequence.

Key Results and Performance

TITAN-Guide demonstrates state-of-the-art performance while significantly lowering hardware requirements.

TITAN-Guide enables various forms of guidance, including aesthetic, audio, and style alignment, without retraining the base model.

Memory Efficiency

The most striking result is the reduction in memory consumption. For a standard 16-frame video (256x256), TITAN-Guide requires only 20.6GB of VRAM, compared to 67.2GB for DOODL and 60.2GB for FreeDoM. This efficiency allows for high-resolution (384x384) guided generation on a single consumer-grade GPU, a task that causes all baseline methods to crash due to out-of-memory (OOM) errors.

Comparison of Fréchet Video Distance (FVD) and Memory Cost across different guidance methods. TITAN-Guide achieves the best (lowest) FVD with the lowest memory footprint.

Quality and Alignment

Using the "Sampled gradient guesses" technique, TITAN-Guide achieved a Fréchet Video Distance (FVD) of 331.11 in audio-video alignment tasks, surpassing DOODL (334.21). Qualitatively, the model successfully aligns video content with specific audio cues (e.g., a car moving when a siren sounds) and transfers complex visual styles (like Van Gogh's "Starry Night") while maintaining text prompt fidelity.

Conclusion

TITAN-Guide provides a practical path for high-quality, guided video generation on accessible hardware. By leveraging forward gradients and partial-frame sampling, it bypasses the prohibitive memory costs of backpropagation. Future work aims to extend this memory-efficient alignment to even more demanding modalities, such as 3D generation and long-form video synthesis.

Summary updated

BibTeX
@inproceedings{simon2025titan,
  author = {Christian Simon and Masato Ishii and Akio Hayakawa and Zhi Zhong and Shusuke Takahashi and Takashi Shibuya and Yuki Mitsufuji},
  title = {{TITAN-Guide: Taming Inference-Time AligNment for Guided Text-to-Video Diffusion Models}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure