Frame Guidance: Training-Free Guidance for Frame-Level Control in Video Diffusion Models

Sangwon Jang, Taekyung Ki, Jaehyeong Jo, Jaehong Yoon, Soo Ye Kim, Zhe Lin, Sung Ju Hwang

TL;DR

Frame Guidance is a training-free, model-agnostic framework that enables fine-grained, frame-level control (such as keyframes, styles, or depth maps) over video diffusion models by guiding only a few selected frames. By introducing "latent slicing" and a stage-aware optimization strategy, it cuts GPU memory usage by up to 60x, allowing complex video steering on a single GPU without any fine-tuning.

What Problem Does This Paper Address?

Controllable video generation—such as forcing a video to match a specific sketch, style, or keyframe sequence—typically requires fine-tuning massive video diffusion models (VDMs). This task-specific fine-tuning is computationally expensive and becomes impractical as models grow and new architectures are frequently released.

While training-free guidance methods exist for images, scaling them to video introduces a massive bottleneck: memory. To compute the guidance gradients for even a single frame, standard methods must decode the entire video latent sequence through complex spatio-temporal VAEs (like CausalVAE). This process easily exceeds 650GB of GPU memory, causing out-of-memory (OOM) errors on standard hardware. The authors address this challenge by creating a general-purpose, training-free guidance method that provides precise frame-level control while remaining highly memory-efficient.

Overview of the Frame Guidance framework showing how gradient-based guidance steers any VDM using sparse frame-level conditions.

Methods

The authors introduce two core components to make training-free video guidance practical:

  1. Latent Slicing: Instead of decoding the entire video latent sequence to calculate the guidance loss for a target frame, the authors decode only a small temporal window (a "slice" of 3 latents). Combined with a 2x spatial downsampling step before decoding, this technique reduces memory overhead dramatically while still providing a strong enough semantic signal to guide the model.
  2. Video Latent Optimization (VLO): Video layouts are largely determined during the early denoising steps, but the high noise levels at this stage can easily wash out standard stochastic guidance signals. VLO introduces a hybrid schedule: it applies deterministic optimization in the early stages to lock in the global layout and motion trajectory, and switches to stochastic optimization (reintroducing noise via a time-travel trick) in the mid-to-late stages to refine fine details and correct accumulated errors.

The framework supports various simple loss functions tailored to different tasks, such as L2 loss for keyframes and depth maps, and Contrastive Style Descriptor (CSD) loss for stylization.

Datasets

The authors evaluated Frame Guidance across several tasks using the following datasets:

  • DAVIS: 40 video clips with at least 81 frames, used to benchmark keyframe-guided video generation and interpolation.
  • Pexels: 30 real-world, highly dynamic, human-centric videos used to test the model's robustness against complex motions and frequent camera changes.
  • StyleCrafter Subset: 6 style reference images paired with 9 distinct content prompts (generating 54 text-style combinations) to evaluate stylized video generation.

Key Insights and Ideas

  • The Reality of CausalVAE's "Temporal Locality": Although CausalVAEs are architecturally designed to enforce strict temporal causality (where past frames influence all future ones), the authors discovered that they actually exhibit temporal locality in practice. Modifying a single frame only significantly impacts a few adjacent latents. This empirical finding is what makes latent slicing possible.
  • Deterministic Updates Lock the Layout: Standard image guidance relies on adding noise back after a gradient update (stochastic). In video, doing this early on destroys the global layout because the noise scale is too large. Deterministic updates are essential in early steps to establish coherent motion.
  • The Denoising Network is the Glue: Coherence across unguided frames is achieved because gradients propagate through the diffusion model's denoising network ($v_\theta$), not the VAE. Bypassing the denoising network to save compute (a common shortcut in image editing) breaks temporal consistency, leading to disconnected videos.

Key Results

  • Massive Memory Savings: Latent slicing combined with spatial downsampling drops peak GPU memory usage from an unmanageable >650GB down to just 41GB for a 14-billion parameter model like Wan2.1, enabling execution on a single commercial GPU.
  • State-of-the-Art Training-Free Performance: On the DAVIS dataset, Frame Guidance applied to CogVideoX achieved a Fréchet Video Distance (FVD) of 577.1, significantly outperforming the training-free TRF baseline (923.1).
  • Outperforming Fine-Tuned Models: When applied to a model already fine-tuned for interpolation (CogX-Interp), Frame Guidance further boosted performance, dropping the FVD from 506.0 to 37.95.
Examples of Frame Guidance applied to keyframe generation, stylization, loop video generation, and structural guidance.

Limitations and Future Work

  • Inference Speed: Because Frame Guidance requires backpropagation through the denoising network and multiple optimization steps per denoising frame, inference is 2x to 4x slower than standard base model generation.
  • Base Model Distribution Dependencies: The framework cannot guide a model to generate content or styles completely alien to its original training data (e.g., it struggles to generate highly out-of-distribution 3D animation styles if the base VDM only knows photorealism).
  • Fine-Grained Structural Control: Loss-based guidance struggles with highly intricate, fine-grained structures (like Sobel edge maps), where training-based control networks (like ControlNet equivalents for video) still offer more stable results. Bounding this structural control remains an open area for future work.

Summary updated

Abstract

Advancements in diffusion models have significantly improved video quality, directing attention to fine-grained controllability. However, many existing methods depend on fine-tuning large-scale video models for specific tasks, which becomes increasingly impractical as model sizes continue to grow. In this work, we present Frame Guidance, a training-free guidance for controllable video generation based on frame-level signals, such as keyframes, style reference images, sketches, or depth maps. For practical training-free guidance, we propose a simple latent processing method that dramatically reduces memory usage, and apply a novel latent optimization strategy designed for globally coherent video generation. Frame Guidance enables effective control across diverse tasks, including keyframe guidance, stylization, and looping, without any training, compatible with any video models. Experimental results show that Frame Guidance can produce high-quality controlled videos for a wide range of tasks and input signals.
BibTeX
@inproceedings{jang2026frame,
  author = {Sangwon Jang and Taekyung Ki and Jaehyeong Jo and Jaehong Yoon and Soo Ye Kim and Zhe Lin and Sung Ju Hwang},
  title = {{Frame Guidance: Training-Free Guidance for Frame-Level Control in Video Diffusion Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=y39XbEp1vK},
}
Figure