Generating video frame-by-frame purely in pixel space is notoriously difficult. Pixels are chaotic, high-dimensional, and lack an inherent understanding of physics, anatomy, or geometry. Without structural constraints, generative models frequently suffer from "noodle limbs," identity drift, and background warping.
To solve this, a powerful paradigm has emerged: guiding generative models using explicit motion proxies—such as 2D skeletons, optical flow, or 3D kinematic parameters. These proxies anchor the synthesis process, acting as an intermediate bridge that enforces physical realism and temporal consistency.
The Evolution of Structural Constraints
Early generative video frameworks leveraged structural proxies to simplify the daunting task of raw pixel prediction. For example, Jacob C Walker et al. demonstrated in The Pose Knows: Video Forecasting by Generating Pose Futures that predicting a person's future skeletal pose first allows a Generative Adversarial Network (GAN) to use that pose as a rigid spatial constraint, maintaining a consistent human appearance during movement.
Similarly, Stephanie Stoll et al. utilized 2D skeletal heatmaps in Sign Language Production using Neural Machine Translation and Generative Adversarial Networks to render specific subjects frame-by-frame without expensive motion capture, while Zezhou Chen et al. mapped low-dimensional facial action vectors to high-resolution images in A Realistic Face-to-Face Conversation System based on Deep Neural Networks to ensure temporal stability in conversational avatars.
With the rise of diffusion models, these structural proxies have transitioned into precise conditioning mechanisms. The authors of Precise Action-to-Video Generation Through Visual Action Prompts showed that injecting 2D skeletons via ControlNet into a pre-trained text-to-video diffusion model provides fine-grained control over complex actions. To scale this to modern architectures, Junyao Gao et al. adapted a Diffusion Transformer (DiT) architecture in CharacterShot: Controllable and Consistent 4D Character Animation, integrating 2D pose sequences directly as a conditioning signal to maintain spatial-temporal consistency across generated frames.
Flow and Depth as Universal Motion Languages
While skeletons are highly effective for human bodies, they fail when animating arbitrary objects or complex camera movements. To overcome this limitation, researchers turned to dense, geometry-aware proxies like optical flow and depth maps.
The authors of MoVideo: Motion-Aware Video Generation with Diffusion Models integrated predicted depth and optical flow maps into a 3D spatio-temporal diffusion model, using these motion cues to guide the synthesis of subsequent frames. This strategy offers a distinct advantage for identity preservation: as Dongnan Liu et al. note in Dance any Beat: Blending Beats with Visuals in Dance Video Generation, synthesizing motion through latent optical flow rather than direct pixel generation preserves the original texture and identity of a subject far more effectively, drastically reducing high-resolution computational complexity.
Furthermore, dense proxies allow for explicit camera manipulation. Qi Dai et al. developed FloVD: Optical Flow Meets Video Diffusion Model for Enhanced Camera-Controlled Video Synthesis, which uses optical flow as a universal language to decouple camera-driven background flow from object motion. This enables precise execution of complex cinematic moves like dolly zooms. Similarly, Jinbo Xing et al. showed in MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generation that translating high-level user intentions into precise 2D motion signals bridges the gap between classical computer graphics principles and modern generative diffusion models.
From Conditional Guidance to Joint Co-Generation
Traditional pipelines treat the motion proxy as a static, pre-computed input condition. However, a major paradigm shift has moved toward joint diffusion, where the video pixels and the structural proxies are generated simultaneously within a unified framework.
Zehuan Huang et al. pioneered this in AnimaX: Animating the Inanimate in 3D with Joint Video-Pose Diffusion Models by simultaneously diffusing multi-view videos and 2D pose sequences. This co-generation process allows rich motion priors inherent in massive video datasets to transfer directly into the controllable domain of skeleton-based animation.
To enforce strict physical laws, models have begun incorporating 3D kinematic parameters directly into the pixel generation loop. Lingwei Dang et al. introduced SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarios, which synchronizes the diffusion of high-fidelity interaction videos and 3D motions. By sharing a unified transformer architecture, the system establishes a closed-loop feedback cycle between 2D visual appearance and 3D physical constraints.
Similarly, Yuxiao Yang et al. developed EchoMotion: Unified Human Video and Motion Generation via Dual-Modality Diffusion Transformer, integrating 3D human body parameters (joint positions and rotations) directly alongside pixel-based latents. By treating motion as a first-class modality, the model learns to enforce kinematic constraints that are typically lost in purely pixel-based objectives.
Implicit Decoders and Low-Fidelity Proxies
As the field matures, researchers are finding ways to inject motion information more seamlessly or with less overhead. The authors of X-Dancer: Expressive Music to Human Dance Video Generation demonstrated that implicitly guiding a diffusion model's latent space with learned motion decoders—rather than explicitly decoding pose skeletons—enables better end-to-end training and smoother integration.
Crucially, steering these models does not always require high-fidelity, perfectly tracked inputs. Assaf Singer et al. introduced a training-free framework in Time-to-Move: Training-Free Motion-Controlled Video Generation via Dual-Clock Denoising, showing that low-fidelity reference animations (such as crude cut-and-paste object translations or depth-based reprojections) provide sufficient layout and motion cues. The diffusion model naturally refines these crude proxies into coherent, high-fidelity visual sequences during the denoising process.
Despite these advances, adapting these motion-sensitive inductive biases to architectures like Diffusion Transformers or 3D-VAEs remains an active frontier. As noted by LEI BAI et al. in A Renaissance of Explicit Motion Information Mining from Transformers for Action Recognition, while explicit motion mining has flourished in discriminative tasks like action recognition, fully unlocking its potential within generative video synthesis continues to be a vital challenge.
Key Foundational Work
Sign Language Production using Neural Machine Translation and Generative Adversarial Networks (2018) — contributed 2 concept elements to this direction.
The Pose Knows: Video Forecasting by Generating Pose Futures (2017) — contributed 1 concept elements to this direction.