Overcoming the Limits of Textual Motion Descriptions

43 concepts41 papers3 proposals

Generative models have achieved remarkable success in synthesizing high-fidelity images and videos from natural language prompts. However, a fundamental bottleneck persists: text is an exceptionally poor medium for describing motion. When a user types "a dog runs in a circle" or "the camera pans smoothly," the underlying model faces an under-specified control problem. Natural language is discrete and symbolic, whereas motion is continuous, highly non-linear, and bound by strict spatial and temporal physics. A single text prompt maps to an infinite manifold of valid trajectories, velocities, and joint configurations.

Faced with this ambiguity, standard video and human motion diffusion models often take the path of least resistance. They either generate nearly static scenes—a failure mode known as "conditional image leakage"—or produce erratic, physically implausible movements. To bridge this textual control precision gap, a major research direction has emerged that moves away from direct text-to-pixel mapping. Instead, researchers are developing architectures that decouple appearance from motion, inject explicit geometric and structural constraints, and model fine-grained part-level kinematics.

The Semantic-Temporal Gap and Image Leakage

To understand why standard models struggle, we must look at how they are trained. Large-scale video datasets often feature simplistic or sparse text descriptions that fail to explicitly detail the underlying motion dynamics. As Bo Dai notes in the development of AnimateDiff, this lack of detailed motion text in training data severely limits a model's ability to align text prompts with specific camera movements or actions without specialized fine-tuning.

Furthermore, when an image-to-video model is conditioned on a high-quality starting frame, it encounters a strong optimization trap. The model can easily minimize training error by simply preserving the spatial structure of the conditioning image across all frames. This tendency frequently overrides the temporal requirements of the text prompt. Shijie Wang et al. address this in MotiF by introducing a Motion Focal Loss. By using optical flow to explicitly identify and prioritize moving regions during training, they force the model to overcome "image leakage" and actually execute the motion described in the text. Without such explicit grounding, as Assaf Singer et al. demonstrate in Time-to-Move, semantic prompts fail to dictate physically plausible trajectories, resulting in unpredictable animations where objects remain static or drift aimlessly.

Decoupling Motion from Appearance

If text cannot precisely dictate motion, the next logical step is to separate what an object looks like from how it moves. By disentangling these two axes, models can ingest coarse text for semantics while utilizing explicit, intermediate representations to guide the dynamics.

One powerful approach is to parse text into intermediate motion fields or optical flows before rendering pixels. For instance, Xiaoyu Shi et al. propose Motion-I2V, a two-stage framework that explicitly decouples the generation process into motion prediction followed by motion-guided rendering. Similarly, Chi Zhang introduces MotionAgent, which uses an agent to parse text prompts into explicit motion representations and composes them into unified optical flow maps to anchor the diffusion process.

This disentanglement can also be learned abstractly. In DisMo, Thomas Ressler-Antal develops abstract motion representations completely independent of an object's identity, allowing for open-world motion transfer. When fine-grained customization is required, Xiuli Bi et al. show in CustomTTT that isolating the "crucial layers" within a video diffusion model allows the system to independently ingest a unique visual subject and a complex reference motion, bypassing the inherent imprecision of natural language prompts.

Structural and Geometric Steering in Latent Space

Instead of relying on the model to infer trajectories from text embeddings alone, how do we directly "steer" the generation process? The answer lies in manipulating the internal structural mechanisms of Diffusion Transformers (DiTs) and U-Nets—specifically their attention layers and positional embeddings.

For camera control, text prompts like "zoom in" or "crane shot" lack the mathematical precision needed for smooth cinematography. To solve this, Hao He et al. introduce CameraCtrl, which parameterizes camera trajectories as geometric Plücker embeddings and injects them directly into the temporal attention layers of video diffusion models. This bypasses text ambiguity entirely, forcing the self-attention mechanism to organize frames around an explicit geometric path.

For object trajectories, Ahmet Berke Gökmen et al. present RoPECraft, a training-free method that warps Rotary Positional Embeddings (RoPE) using optical flow. Because positional embeddings dictate where features "belong" in space and time, warping them directly forces the model to shift its internal representations along a specified trajectory. In a similar vein, Alexander Pondaven et al. develop Video Motion Transfer with Diffusion Transformers, extracting an "Attention Motion Flow" signal from a reference video's internal attention maps to steer a new generation, proving that motion can be transferred directly via the model's internal routing mechanisms without relying on text descriptions.

Part-Level Kinematics and Compositional Control

The limits of textual description become even more severe when animating complex structures like the human body or scenes with multiple interacting entities. A single phrase like "the person dances" treats the entire skeleton as a single unit, making it impossible to command specific limbs to perform independent or hybrid actions.

To achieve fine-grained control, researchers have introduced hierarchical and part-level modeling. Zhenguang Liu et al. address this in Aggregated Multi-GANs for Controlled 3D Human Motion Prediction by breaking down the skeleton to allow independent control over individual body parts, enabling composite movements. For text-driven synthesis, Mingyuan Zhang et al. introduce MotionDiffuse, a diffusion framework that enables fine-grained motion synthesis by allowing independent control over individual body segments. The authors of KinMo take this further by incorporating explicit kinematic descriptions of body parts, bridging the semantic-temporal gap by aligning local joint movements with hierarchical text structures.

When spatial and temporal requirements are highly specific—such as a hand touching a precise 3D coordinate at an exact frame—Yiming Xie et al. demonstrate with OmniControl that models can accept flexible joint-level constraints at any time step. Finally, when scenes involve multiple entities, standard foundation models often fail to coordinate their movements due to compositional text ambiguity. As Leigang Qu et al. show in TTOM, optimizing the model's latents at test-time helps enforce precise numeracy, spatial positioning, and coordinated motion that text prompts alone fail to guarantee.

Key Foundational Work

Motion Capture Synthesis with Adversarial Learning (2017) — contributed 1 concept elements to this direction.
Aggregated Multi-GANs for Controlled 3D Human Motion Prediction (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_textual_motion_description_limits_a82791be,
  title = {textual motion description limits},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/textual-motion-description-limits/rdr_a82791be866c4007bdd9e6c2c15220f7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_textual_motion_description_limits_a82791be_v1,
  title = {textual motion description limits},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/textual-motion-description-limits/rdr_a82791be866c4007bdd9e6c2c15220f7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake