Compressing Trajectories into Continuous Latent Action Spaces

48 concepts46 papers

When training an agent to perform physical tasks—whether it is a quadruped robot navigating rough terrain or a manipulator folding dough—the standard approach is to operate directly within the raw action space. This means the controller must output precise joint torques, steering angles, or velocity vectors at every single time step, often at frequencies of 60 Hz or higher.

This raw action space presents a fundamental challenge: the curse of dimensionality. Exploring at the level of high-frequency motor commands is incredibly inefficient because the vast majority of possible command sequences result in useless jitter, physical stalls, or destructive behaviors. To solve this, a growing body of research focuses on trajectory compression, a technique that maps high-dimensional, temporally extended sequences of actions into a continuous, low-dimensional latent space. By navigating this compressed manifold, agents can plan and learn at the level of behavioral skills rather than raw motor primitives.

The Trajectory Autoencoder as a Physical Filter

The core mechanism behind this approach relies on an encoder-decoder architecture, typically a Variational Autoencoder (VAE). Instead of processing individual actions in isolation, the framework operates on action chunks or entire trajectories.

During a task-agnostic pre-training phase, the model is fed a rich dataset of valid movements, which can come from human demonstrations, simulated play data, or biological observations. The encoder compresses these multi-joint, time-varying trajectories into a compact latent vector $z$. The decoder is then tasked with reconstructing the original sequence of control commands from this single point.

[High-Dimensional Trajectory] ---> (Encoder) ---> [Low-Dimensional Latent Space (z)]
                                                         |
                                                         v
[Smooth Control Commands]    <--- (Decoder) <------------+

Why does this make control so much easier? The decoder effectively acts as both a spatial and a temporal filter. As shown by Wang et al., embedding high-frequency action sequences into a latent space ensures that reconstructed actions maintain physical consistency and smoothness, eliminating the "stop-and-go" jitter common in step-by-step controllers.

By pre-training this manifold in simulation, as demonstrated by Ghadirzadeh et al., the system maps out the underlying structure of valid motions. The latent space becomes a continuous landscape where every single coordinate represents a physically plausible, smooth trajectory. High-level controllers can then steer the agent's behavior by navigating this simplified manifold rather than managing individual joint torques directly, a strategy that Shi et al. leveraged to bypass reinforcement learning entirely for agile motor skills.

This representational strategy is not limited to robotics; it mirrors how we understand biological motion. For instance, Sharma et al. utilized latent variable models to map high-dimensional zebrafish postural data into discrete swim bouts modulated by internal states, while van Dam et al. used recurrent VAEs to isolate essential rodent behavioral dynamics from high-frequency execution noise.

Why It Works: The Geometry of Skill Priors

When a reinforcement learning (RL) agent or a motion planner operates in a raw action space, its exploration is entirely unguided. It must discover the laws of physics and joint coordination through trial and error.

By shifting the optimization problem to a compressed latent space, the agent inherits a skill prior. As argued by Pertsch et al., compressing action sequences into a continuous latent space allows a high-level controller to explore a dense space of reusable behaviors rather than a sparse space of primitive actions. If the agent wants to turn left, it does not need to coordinate dozens of high-frequency joint adjustments; it simply shifts its position in the latent space toward the "turn left" coordinate.

This geometric simplification enables highly efficient planning and optimization:

  • Gradient-Based Planning: Because the latent space is continuous and differentiable, planners can use gradient descent directly within the latent space to find optimal trajectories. Yamada et al. demonstrated this by combining scene embeddings with latent spaces to achieve real-time motion planning in dynamic environments.
  • Complex Deformable Manipulation: For tasks with high degrees of freedom, such as manipulating cloth or dough, Yamada et al. showed that latent diffusion models operating on compressed motor commands keep generated actions physically plausible over long horizons.
  • Handling Multi-Modality: Demonstration data often contains multiple ways to solve a single problem. Tosatto addressed this by mapping movement parameters into a latent space using a mixture of probabilistic principal component analyzers, capturing distinct behavioral modes without averaging them into unrealistic trajectories.

Structuring and Steering the Latent Space

Once a latent action space is established, the next challenge is providing an intuitive interface for either an autonomous policy or a human operator to steer it.

One powerful method is to inject structured geometric or periodic signals directly into the latent space. For example, quadrupedal locomotion is inherently rhythmic. Mitchell et al. showed that by injecting a simple oscillatory drive signal into a gait-compressing VAE, a robot can continuously vary its speed and step height without needing discrete contact schedules. Expanding on this concept, Jones et al. demonstrated that both periodic walking and non-periodic reaching can be controlled by overwriting specific latent dimensions with simple geometric signals, creating an interpretable interface for high-degree-of-freedom systems.

[Oscillatory/Geometric Signal] ---> [Targeted Latent Dimension] ---> (Decoder) ---> [Complex Periodic Motion]

This abstraction also simplifies human-robot collaboration. In shared autonomy frameworks, such as the LILAC system developed by the authors of No, to the Right: Online Language Corrections for Robotic Manipulation via Shared Autonomy, low-dimensional user inputs like joystick movements are mapped directly into a high-dimensional control manifold. When the user gives a verbal correction, the system dynamically warps the latent space, allowing the operator to guide intricate tasks without manually managing every degree of freedom.

Distilling the Essence of Motion

A final advantage of latent action mapping is its ability to isolate the core intent of a movement from environmental or embodiment noise. When learning from uncurated sources, such as internet videos, models can easily overfit to static backgrounds or specific robot body shapes.

To combat this, Yang et al. introduced an information bottleneck to latent motion representations, preventing the encoding of static appearance noise while preserving critical motion details. Similarly, Torr demonstrated that tuning the dimensionality of a task-relevant embedding acts as a granularity control for cross-domain imitation: a tight bottleneck forces an agent to copy only high-level goals, while a wider space allows for the transfer of stylistic details across entirely different physical bodies.

By transforming the problem of control from a high-frequency sequence of raw numbers into a low-dimensional trajectory through a structured skill manifold, latent action mapping provides the mathematical bridge necessary for sample-efficient, robust, and interpretable physical AI.

Key Foundational Work

Deep Predictive Policy Training using Reinforcement Learning (2017) — contributed 1 concept elements to this direction.
Point process latent variable models of larval zebrafish behavior (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_latent_action_mapping_94d29a7e,
  title = {latent action mapping},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/latent-action-mapping/rdr_94d29a7e6c4f482797bd3db1fbe2ee9f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_latent_action_mapping_94d29a7e_v1,
  title = {latent action mapping},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/latent-action-mapping/rdr_94d29a7e6c4f482797bd3db1fbe2ee9f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake