When a human transitions from a sprint to a crouch-walk, or blends a wave into a stride, the movement feels fluid and natural. In contrast, traditional robotic controllers and animation systems often treat behaviors as discrete, isolated primitives. If you attempt to blend two distinct actions—such as a walk and a jump—by simply averaging their raw joint angles over time, the result is often a physically impossible, kinematically broken motion.
To solve this, a powerful paradigm has emerged: mapping high-dimensional behaviors into a continuous latent manifold. By compressing complex trajectories, control policies, or skeletal sequences into a low-dimensional space, the underlying physics and semantics of motion are preserved geometrically. In this latent space, drawing a straight line between two distinct behaviors does not produce a broken artifact; instead, it yields a smooth, physically plausible intermediate action.
The Geometry of Movement
Why does compressing a behavior into a latent space make it interpolatable? The intuition lies in how generative models organize data. When a model is trained to compress and reconstruct diverse behaviors, it must find a compact way to represent variations in style, speed, and direction.
Raquel Urtasun et al. demonstrated this structural organization in 3D People Tracking with Gaussian Process Dynamical Models, showing that low-dimensional latent spaces for cyclic motions naturally organize into topological loops. Crucially, the geometric properties of these loops—such as their size—correspond directly to physical variations like movement speed.
To ensure that every point in the latent space corresponds to a valid behavior, researchers use probabilistic and variational frameworks to enforce density and smoothness. For instance, Victor Campos 0001 et al. showed in Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills that modeling the relationship between states and skills using a variational framework creates a well-structured space capable of generating novel behaviors via interpolation. Similarly, Chen Tessler et al. leveraged alignment and uniformity constraints in CALM: Conditional Adversarial Latent Models for Directable Virtual Characters to cluster similar behaviors together, allowing a virtual character to transition seamlessly from a sprint to a crouch-walk.
Why Latent Interpolation Preserves Physics
To understand why interpolation works in a latent space but fails in raw joint space, consider the difference between a flat grid and a curved surface. Raw joint configurations exist in a high-dimensional space filled with invalid states (e.g., joints bending backward or feet clipping through the floor). A learned latent space acts as a dimensionality reduction mechanism that isolates the true, low-dimensional manifold of valid human or robot configurations.
As Sethu Vijayakumar et al. observed in Does dimensionality reduction improve the quality of motion interpolation?, performing interpolation within this reduced space ensures that the generated intermediate poses remain strictly within the physically valid configuration space. Pei Lv et al. reinforced this in Rhythmical motion synthesis based on Gaussian process dynamical model, noting that manipulating motion within a learned dynamical latent representation respects the constraints of the natural motion manifold, preventing the robotic artifacts associated with direct joint manipulation.
For highly complex maneuvers, the shortest path between two behaviors might not be a straight line (linear interpolation), but a curve that follows the contours of the manifold. Hadi Beik-Mohammadi et al. formalized this in Learning Riemannian Manifolds for Geodesic Motion Skills by using geodesic pathfinding on a learned manifold to synthesize novel, valid behaviors that smoothly blend demonstrated modes of a task.
Raw Joint Space (Linear Blending) Learned Latent Manifold (Geodesic Path)
[Pose A] [Pose A]
\ / \
(Invalid State) / \
/ / [Valid Hybrid]
[Pose B] / \
[Pose B]--------'
Compositionality and Zero-Shot Generalization
Once a continuous behavioral manifold is established, it unlocks the ability to synthesize entirely new actions without explicit retraining. High-level planners or reinforcement learning agents can navigate this space to discover "in-between" gaits or combine distinct skills.
Blending Primitives
John D. Co-Reyes et al. demonstrated in Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings that a continuous latent space allows an agent to smoothly vary the distance or direction of an object's movement by selecting intermediate vectors. This capacity to fill the gaps between discrete training examples enables robust adaptation; for example, Roberto Calandra et al. showed in Planning in Learned Latent Action Spaces for Generalizable Legged Locomotion that interpolating between discrete motion primitives allows a legged robot to discover optimal intermediate gaits to counteract environmental disturbances.
Vector Arithmetic and Chords
Because the latent space is continuous and structured, behaviors can be combined using vector arithmetic. Andre Barreto et al. introduced this concept in The Option Keyboard: Combining Skills in Reinforcement Learning, demonstrating that an agent trained on a sparse set of basis skills can navigate in any arbitrary direction by blending those basis vectors. More recently, Shenyuan Gao et al. showed in AdaWorld: Learning Adaptable World Models with Latent Actions that averaging the latent action vectors of two distinct movements allows a world model to simulate the simultaneous execution of both actions.
Semantic Alignment
By aligning these motor manifolds with text embeddings, we can control physical agents using natural language. Sanja Fidler argued in PADL: Language-Directed Physics-Based Character Control that structuring the motor control space around semantically aligned embeddings enables the generation of novel behaviors through latent space interpolation. Blending the embeddings for distinct text commands produces physically plausible intermediate motions—such as a crouching walk—even if that specific behavior was completely absent from the training data.
Applications Beyond Robotics
The intuition of continuous behavioral interpolation extends far beyond robot locomotion, proving highly effective across computer vision, animation, and object interaction:
- Articulated Object Manipulation: Haowen Wang demonstrated in PD$^{2}$GS: Part-Level Decoupling and Continuous Deformation of Articulated Objects via Gaussian Splatting that representing articulated motion as a continuous latent manifold enables the generation of physically plausible intermediate states, preventing geometric artifacts like part overlapping when animating objects like laptops or drawers.
- Video Prediction: Kevin J. Shih showed in Video Interpolation and Prediction with Unsupervised Landmarks that coordinate-based latent spaces facilitate smooth interpolation between distant video frames by allowing linear transitions in a structured landmark space, preserving the geometric integrity of moving objects.
- Visual Speech Synthesis: Anna Hilsmann leveraged this concept in Neural Face Models for Example-Based Visual Speech Synthesis, performing interpolation within a learned continuous latent manifold to generate intermediate facial frames that maintain realistic dynamics and eliminate the visual artifacts common in pixel-space transitions.
- Motion Capture: Andrea Fossati showed in Bridging the Gap between Detection and Tracking for 3D Monocular Video-Based Motion Capture that mapping 3D capture data into a low-dimensional eigen-space constrains reconstructed motion to a learned manifold of valid postures, ensuring kinematic realism even when tracking through severe occlusions.
By transforming the discrete, high-dimensional problem of behavioral control into a smooth, low-dimensional geometric problem, latent skill interpolation provides a foundational mechanism for fluid, sample-efficient, and generalizable intelligence.
Key Foundational Work
3D People Tracking with Gaussian Process Dynamical Models (2006) — contributed 1 concept elements to this direction.
Reinforcement Learning in Continuous State and Action Space (2003) — contributed 1 concept elements to this direction.