Early text-to-video models operated like a "black box": you provided a prompt, and the model returned a video that might or might not match your mental image. As the field matures, research has shifted from simple generation to fine-grained controllability. Instead of relying on text as a blunt instrument, researchers are developing modular frameworks that allow users to steer motion, identity, and structure independently.
The core intuition behind this shift is decoupling: separating the "what" (content and identity) from the "how" (motion and trajectories). By treating a pre-trained video diffusion model as a frozen "generative prior," researchers can layer specialized control modules on top without the prohibitive cost of retraining the entire system.
Decoupling Motion from Identity
A recurring challenge in video generation is "appearance leakage," where the motion of an object inadvertently changes its identity—for example, a car turning into a bus as it moves. To solve this, frameworks like DisMo by Thomas Ressler-Antal and colleagues focus on learning abstract motion representations that are entirely independent of an object's appearance.
This disentanglement allows for "open-world" motion transfer. In ConMo, Jiayi Gao demonstrates that by "diluting" motion cues with background data, one can transfer a car’s specific trajectory to a running dog. The system understands the path of the motion without being tethered to the shape of the original object. Similarly, VMC, developed by Hyeonho Jeong and others, isolates motion by fine-tuning only the temporal attention layers, ensuring the model learns how things move while keeping their appearance fixed.
Steering with Trajectories and Blueprints
If text is too vague, what is the ideal control signal? Many researchers have converged on sparse trajectories—simple lines or points that "conduct" the movement on screen.
- 2D to 3D Paths: While many tools use 2D drag-and-drop interfaces, LeviTor (and its CVPR 2025 iteration by Hanlin Wang) introduces 3D trajectory control. By representing objects as clusters of points with depth, the model can handle complex movements like orbiting or perspective-correct scaling that 2D methods miss.
- Interactive Interpolation: Framer, by Wen Wang and others, applies this to frame interpolation. Instead of the AI guessing how to get from Image A to Image B, the user provides keypoint trajectories to resolve the ambiguity of the transition.
- Hierarchical Planning: For longer videos, the control signal becomes a "blueprint." ARLON, by Zongyi Li, uses an autoregressive model to generate a coarse spatial-temporal sketch, which then guides a high-resolution Diffusion Transformer (DiT) to fill in the details.
The Adapter Pattern: Efficiency through Modularity
Training a video model from scratch is computationally ruinous. The dominant architectural pattern is now the Adapter—a lightweight layer that bridges a control signal to a frozen backbone.
Han Lin and the team behind CTRL-Adapter demonstrated that you can reuse ControlNets originally built for image models (like Stable Diffusion) and adapt them to video backbones in under 10 GPU hours. This modularity is further refined in LiON-LoRA, which uses linear scalability and orthogonality to fuse multiple controls (like camera movement and object motion) without them "fighting" each other.
The Generative Ceiling
Across nearly 200 papers, a singular, sobering conclusion emerges: the control mechanism is only as good as the backbone it steers.
Whether it is George Stoica distilling structural knowledge in SAM2VideoX or Wei Gao using LLMs to enforce physics in PhyT2V, the "generative prior" acts as a hard ceiling. If the underlying model (like SVD or CogVideoX) hasn't learned how to render a human hand or a complex fluid texture, no amount of precise trajectory guidance or prompt refinement can force it to do so.
As Yuwei Guo notes in SparseCtrl, these control modules are "inherently limited by the motion priors and generative quality of the underlying backbone." Consequently, the next frontier in controllable video isn't just better adapters, but more capable foundation models that provide a richer palette for these controllers to act upon.