Modern video generation models can synthesize remarkably photorealistic frames, yet they often struggle with precise spatial control. A text prompt like "a car turning left" tells a model what to generate, but it cannot specify the exact geometric arc, velocity, or pixel-level path of the vehicle. Relying solely on text or stochastic sampling frequently results in generic, unpredictable motion patterns.
To bridge this gap, a prominent research direction focuses on trajectory guidance—conditioning generative models on explicit spatial-temporal paths (such as bounding boxes, point tracks, or camera coordinates). By integrating geometric trajectories directly into the generative pipeline, these methods transform video synthesis from a passive prediction task into a highly steerable, interactive simulation.
Guiding the Latent Space and Attention Maps
To understand why trajectory guidance works, we must look at how video diffusion models construct scenes. During the early stages of the denoising process, the model establishes the layout and structural composition of a video through its internal attention layers. Trajectory guidance exploits this mechanism by forcing the model's internal representations to align with user-defined coordinates over time.
One approach achieves this without any additional training or fine-tuning. In Peekaboo: Interactive Video Generation via Masked-Diffusion, Anshul Nasery et al. demonstrate that applying specialized spatio-temporal masks directly to the model's internal attention layers during the initial generation steps forces the model to bind specific objects to designated paths. Similarly, Changgu Chen et al. introduce Motion-Zero: A Zero-Shot Trajectory Control Framework of Moving Object for Diffusion-Based Video Generation, utilizing bounding boxes to dynamically manipulate attention maps at test-time, ensuring objects follow precise trajectories while preserving high visual quality.
[User Trajectory/Bounding Box]
│
▼
[Spatio-Temporal Attention Masking] ──► [Forces Feature Clustering along Path] ──► [Steered Video Output]
When training is an option, researchers can embed trajectory awareness deeper into the network architecture. Rather than relying on complex auxiliary encoders that can degrade generation quality, Ruihang Chu et al. present Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance. The authors show that directly propagating latent features along specified trajectory vectors provides a highly precise motion control mechanism, ensuring that the underlying data structures physically shift in accordance with the user's input.
Decoupling Motion from Appearance via Flow Fields
Directly generating pixels from a trajectory is a difficult optimization problem because the model must simultaneously figure out what the scene looks like (appearance) and how the scene moves (geometry). To simplify this task, many frameworks decouple the process by introducing intermediate motion representations, such as optical flow or dense point grids.
By predicting or conditioning on these intermediate structures, the model anchors its generation to a physically plausible geometric backbone. For instance, Gabrijel Boduljak et al. demonstrate in What Happens Next? Anticipating Future Motion by Generating Point Trajectories that generating dense grids of point trajectories directly from a single image serves as an excellent foundation for video synthesis, stripping away pixel overhead to focus purely on physical consistency.
Once these motion paths are established, they act as a rigid constraint for the diffusion process. In FLIP: Flow-Centric Generative Planning for General-Purpose Manipulation Tasks, Chongkai Gao et al. utilize pixel-level image flow as a universal language to condition video generation, allowing a world model to accurately simulate how objects react to physical interactions. This concept of isolating motion cues is echoed by Zhicheng Zhang et al. in ExtDM: Distribution Extrapolation Diffusion Model for Video Prediction, where explicitly modeling motion cues as intermediate distributions allows the generative process to extrapolate future frames along a predefined path without losing visual fidelity.
Differential Guidance and Camera Kinematics
Controlling an individual object's trajectory is only half the battle; controlling the camera's trajectory introduces a different set of challenges. Because a camera movement shifts every single pixel in the frame simultaneously, unguided models often suffer from perspective drifting, jitter, or structural tearing.
To solve this, researchers adapt the principles of classifier-free guidance (CFG)—traditionally used to amplify text prompts—and apply them to motion vectors. In Boosting Camera Motion Control for Video Diffusion Transformers, Soon Yau Cheong et al. introduce Camera Motion Guidance (CMG). This technique defines a completely static view as a "null condition." By calculating the mathematical difference between the target camera trajectory and this static reference baseline, the system can dramatically amplify the intensity and accuracy of camera movements during the diffusion process.
Classifier-Free Motion Guidance:
Directional Signal = Target Trajectory Condition - Static View (Null Condition)
Furthermore, true cinematic control requires a deep understanding of the relationship between the camera and the subject. In E.T. the Exceptional Trajectories: Text-to-camera-trajectory generation with character awareness, Robin Courant et al. demonstrate that conditioning camera trajectory generation directly on the 3D path of the subject allows models to learn complex, synchronized cinematic behaviors (like tracking or panning shots).
To ensure these long-form environmental sequences remain stable over time, frameworks like the one proposed by Chuanhao Li et al. in Sekai: A Video Dataset towards World Exploration integrate per-frame camera poses directly as a continuous conditioning signal during training. This transforms video synthesis into an active navigation task, allowing users to steer the camera along explicit spatial paths while maintaining total environmental consistency.
Key Foundational Work
Machine Learning for Video-Based Rendering (2000) — contributed 1 concept elements to this direction.
SceneNet RGB-D: Can 5M Synthetic Images Beat Generic ImageNet Pre-training on Indoor Segmentation? (2017) — contributed 1 concept elements to this direction.