Frame Guidance: Training-Free Guidance for Frame-Level Control in Video Diffusion Models

Sangwon Jang, Taekyung Ki, Jaehyeong Jo, Jaehong Yoon, Soo Ye Kim, Zhe Lin, Sung Ju Hwang

Introduction

Advancements in diffusion models have significantly improved video quality, shifting the research focus toward fine-grained controllability. However, most existing methods rely on fine-tuning large-scale models for specific tasks—an approach that is increasingly impractical as model sizes grow. Frame Guidance introduces a training-free, model-agnostic framework that enables precise control over video generation using simple frame-level signals like keyframes, sketches, or style references.

Frame Guidance enables diverse control tasks such as stylization and keyframe-based video generation without model training.

The Challenge: Memory and Coherence

Controllable video generation typically requires specialized datasets and expensive fine-tuning (e.g., for depth-to-video). Training-free alternatives often face two major hurdles:

  1. Memory Constraints: Calculating guidance loss usually requires decoding the entire video latent sequence. For high-resolution videos, this can demand hundreds of gigabytes of VRAM, far exceeding the capacity of standard GPUs.
  2. Temporal Incoherence: Forcing specific frames to match inputs often results in flickering or the model ignoring the guidance entirely because the global layout isn't established correctly during the early, noisy stages of diffusion.

Technical Innovations

Frame Guidance addresses these issues through two core innovations:

Latent Slicing

The researchers identified a property called temporal locality in CausalVAEs. Despite being designed for long-term dependencies, a single frame's latent primarily influences only a small window of neighboring frames. By decoding only a "slice" (e.g., 3 latents) instead of the full sequence, memory usage is reduced by up to 60x, allowing the system to run on a single H100 or smaller consumer-grade GPUs.

Video Latent Optimization (VLO)

Standard guidance often fails because stochastic updates "wash out" control signals early on. VLO employs a hybrid strategy:

  • Deterministic Updates: Applied in early diffusion steps to "lock in" global layout and motion.
  • Stochastic Updates: Used in later steps to refine details and maintain texture.
Keyframe guidance allows for precise temporal control by providing specific frames at designated time steps.

Key Insights

  • Locality is Key: The discovery of high temporal locality in CausalVAEs fundamentally changes how we approach backpropagation in video models, proving that global decoding is unnecessary for local frame guidance.
  • Layout vs. Detail: The "skeleton" of a video is determined in the first few steps of diffusion. Guidance must dominate these early steps to ensure the model adheres to user constraints.
  • Guidance Propagation: Guiding just a few anchor frames (like the first and last) is sufficient. The denoising network naturally propagates this gradient information across the temporal dimension, ensuring intermediate frames adjust to match the anchors.

Experimental Results

Frame Guidance was evaluated across benchmarks like DAVIS and Pexels for keyframe guidance, and StyleCrafter for stylization.

  • Performance: It outperformed training-free baselines and even some fine-tuned models in terms of FVD and FID, indicating superior visual quality and consistency.
  • Efficiency: On models like CogVideoX-5B, the method fits within standard GPU limits, whereas naive guidance would require over 650GB of VRAM.
  • Versatility: The framework enables "impossible" tasks, such as turning coarse color blocks into photorealistic videos or generating seamless looping videos by enforcing consistency between the first and last frames.

Limitations and Future Work

While powerful, Frame Guidance is 2x to 4x slower than standard generation due to the required backpropagation. Additionally, it cannot "teach" a model styles it hasn't seen before. Future work will focus on improving the influence of sparse structural signals, like thin edge maps, which currently provide weaker control than dedicated ControlNets.

Summary updated

Abstract

Advancements in diffusion models have significantly improved video quality, directing attention to fine-grained controllability. However, many existing methods depend on fine-tuning large-scale video models for specific tasks, which becomes increasingly impractical as model sizes continue to grow. In this work, we present Frame Guidance, a training-free guidance for controllable video generation based on frame-level signals, such as keyframes, style reference images, sketches, or depth maps. For practical training-free guidance, we propose a simple latent processing method that dramatically reduces memory usage, and apply a novel latent optimization strategy designed for globally coherent video generation. Frame Guidance enables effective control across diverse tasks, including keyframe guidance, stylization, and looping, without any training, compatible with any video models. Experimental results show that Frame Guidance can produce high-quality controlled videos for a wide range of tasks and input signals.
BibTeX
@inproceedings{jang2026frame,
  author = {Sangwon Jang and Taekyung Ki and Jaehyeong Jo and Jaehong Yoon and Soo Ye Kim and Zhe Lin and Sung Ju Hwang},
  title = {{Frame Guidance: Training-Free Guidance for Frame-Level Control in Video Diffusion Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=y39XbEp1vK},
}
Figure