Early Stage Motion Guidance in Video Diffusion

25 concepts18 papers1 proposal

When a video diffusion model generates a sequence of frames, it does not resolve layout, motion, and fine textures simultaneously. Instead, it operates under a strict temporal hierarchy. In the early, high-noise stages of the denoising process, the model establishes global layout, camera trajectories, and coarse motion paths. Only in the later, low-noise stages does it fill in high-frequency visual details like textures and fine structural edges.

This insight has sparked a powerful research direction: by scheduling motion guidance, structural constraints, and temporal attention manipulations exclusively during the initial stages of the denoising process, we can dramatically improve temporal consistency and motion control without retraining the underlying models.

The Temporal Decoupling of Motion and Appearance

The core reason early-stage guidance works is that motion and appearance are naturally decoupled across the diffusion timeline. As observed by Liu et al., motion modeling primarily occurs during the initial high-noise timesteps. During these early steps, the latent space is highly fluid, and the model acts as a low-pass filter, mapping out where objects will move before deciding what they look like.

If we attempt to apply motion or structural guidance throughout the entire generation process, the results degrade. For instance, Skorokhodov et al. found that restricting 3D camera conditioning to the early stages of generation actually improves final video quality. Because global camera motion is a low-frequency feature determined early on, forcing the model to process camera signals in later steps is redundant and actively disrupts the generation of high-frequency visual details.

Similarly, in the context of video subject swapping, Gu et al. demonstrated that applying semantic point correspondence guidance primarily during the initial stages optimizes the high-level layout and motion alignment. Restricting guidance to this window prevents the model from erroneously bleeding low-level textures or fine-grained details from the source video into the new subject.

Mechanisms of Early-Stage Intervention

Because the fundamental motion trajectory is locked in early, researchers have designed a variety of training-free, "plug-and-play" interventions that target this critical window—typically the first 20% to 40% of the denoising process.

Attention and Feature Regulation

In text-to-video models, structural glitches often propagate if the initial layout is unstable. To address this, Bu et al. introduced a method to manipulate temporal attention maps exclusively during the first 20% of diffusion steps, fixing structural errors before they can manifest. In a subsequent work, Bu et al. amplified the high-frequency energy of these attention maps during the same early window to breathe life into static scenes.

For complex, multi-step text prompts, Zhang et al. utilized temporal-wise separable attention during the initial 20% to 40% of denoising. This allows precise control over when and where specific actions occur in the video timeline without modifying the fine-grained texture generation that happens later.

Trajectory Anchoring and Variance Reduction

When generating long videos, maintaining long-term consistency is a major challenge. Lu et al. selectively applied feature fusion during the initial phases of the denoising trajectory to anchor the global scene composition, leaving later steps free to refine local temporal details.

To eliminate physical implausibilities like disappearing objects or extra limbs, Shaulov et al. developed a method that identifies and minimizes high-variance "jitter" in internal latent representations. By restricting this optimization to the early-to-mid window of generation, they smooth out global motion trajectories while preserving fine textures.

Furthermore, Xiao et al. isolated internal "motion features" (MOFT) within video diffusion models, proving that motion guidance is highly effective early in the timeline, whereas later stages are strictly better suited for refining spatial consistency and pixel details.

Optimizing the Sampling Path

Recognizing that early timesteps dictate global structure allows researchers to modify the sampling path itself to favor motion coherence.

One approach is to bias the entire sampling distribution. Zheng demonstrated that biasing the diffusion process toward high-noise timesteps forces the model to prioritize large-scale motion and global structural changes over fine textures, resulting in more realistic and coherent movement sequences.

Another approach uses teacher models or causal guidance to steer the early path. Lee et al. showed that intervening only during the initial steps with a high-performing "teacher" model is sufficient to establish global structure and motion consistency in a simpler "student" model. To generate long, multi-action video narratives without retraining, Huang et al. applied complex causal path-integral guidance exclusively during the initial high-noise steps. This achieves the temporal smoothness of natively trained long-video architectures while minimizing inference-time overhead.

Resolving Motion Conflicts in Inbetweening

The principle of early-stage motion determination is also vital for generative video inbetweening—creating smooth frames between a starting and ending image. Standard bidirectional diffusion models often suffer from "motion conflicts" where the forward and backward denoising paths disagree, causing ghosting or jitter.

To resolve this, Jeon et al. applied motion constraints exclusively during the early, high-noise stages to distill a unified trajectory from both paths. In later stages, the system reverts to standard sampling so the model can focus on exact frame matching and visual fidelity. Similarly, Zhang et al. demonstrated that aligning conflicting motion priors along a consensus axis during the first 20% of the denoising process is entirely sufficient to eliminate artifacts, leaving the remaining steps untouched for detail refinement.

Shaping Motion at Step Zero

The ultimate extension of early-stage motion scheduling happens before the denoising process even begins: at the initial noise state (timestep $T$).

Instead of starting from standard independent Gaussian noise, Burgert et al. structured the initial latent noise according to optical flow fields. This allows the model to treat motion as a pre-defined prior, establishing temporal paths before denoising begins. In a similar vein, Xu et al. initialized diffusion models with noise derived from the temporal structure of retrieved reference videos. By grounding the initial noise in real-world dynamics, the model is guided toward realistic motion paths from the very first step.

Key Foundational Work

FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generation (2025) — contributed 2 concept elements to this direction.
Denoising Reuse: Exploiting Inter-Frame Motion Consistency for Efficient Video Generation (2025) — contributed 2 concept elements to this direction.
FreeAction: Training-Free Techniques for Enhanced Fidelity of Trajectory-to-Video Generation (2025) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_coarse_to_fine_motion_scheduling_b80d4a44,
  title = {coarse-to-fine motion scheduling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/coarse-to-fine-motion-scheduling/rdr_b80d4a447aee42b3bdace49280c3e68b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_coarse_to_fine_motion_scheduling_b80d4a44_v1,
  title = {coarse-to-fine motion scheduling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/coarse-to-fine-motion-scheduling/rdr_b80d4a447aee42b3bdace49280c3e68b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake