Unified Latent Action Spaces for Cross-Embodiment Robot Learning

22 concepts15 papers

The Embodiment Barrier

In traditional robotics, a control policy is tightly coupled to a specific machine. A policy trained to control a 7-degree-of-freedom industrial arm cannot naturally operate a bimanual robot, let alone a humanoid or a drone. This is because their action spaces—the raw joint torques, angles, or velocities—are fundamentally incompatible.

This incompatibility creates a massive data bottleneck. While computer vision and natural language processing benefit from massive, internet-scale datasets, robotics models are typically confined to small datasets collected on the exact hardware they are meant to control. Furthermore, it prevents robots from learning from the most abundant source of behavioral data on the planet: videos of humans. When a human demonstrates a task, there are no recorded joint torques or digital control signals; there are only changing pixels.

To build true foundation models for robotics, we must break this embodiment barrier. The solution emerging across recent research is to decouple what an action intends to achieve from how a specific body executes it, by mapping heterogeneous control signals into a unified latent action space.

The Core Intuition: Functional Invariance

Why should a unified action space even be possible across completely different physical forms? The key insight is that while the physical execution of a task depends heavily on an agent's morphology, the functional intent and the resulting environmental transitions are invariant.

Consider the act of opening a drawer. A human hand pulls a handle using complex muscular coordination; a two-fingered robotic gripper pinches and pulls; a vacuum suction cup adheres and retracts. The low-level control signals look entirely different. However, from the perspective of the world, the outcome is identical: the drawer transitions from closed to open.

By forcing a model to represent actions based on their semantic intent or their causal impact on the environment, we can construct a shared mathematical language. As explored by Huang et al. in Cross-Embodiment Robot Foundation World Models with Latent Actions, mapping diverse robotic control signals into a continuous latent domain based on their semantic impact allows generalist models to treat functionally similar movements as equivalent across different physical embodiments.

[Human Video] ---------> ( Semantic / Functional ) <--------- [Robot Joint Torques]
[Bimanual Robot] -------> (  Latent Action Space  ) <--------- [Gripper Velocities]
                                   |
                                   v
                       [Downstream Robot Control]

Mathematical Mechanics of Alignment

To build this shared manifold, researchers employ several distinct alignment strategies that force disparate hardware commands into a unified representation.

Contrastive Alignment

One of the most direct ways to enforce functional equivalence is through contrastive learning. As demonstrated by Nava et al. in Latent Action Diffusion for Cross-Embodiment Manipulation, contrastive learning objectives can be used to group actions with the same semantic intent together in the latent space, regardless of the hardware configuration. By maximizing the representational similarity of behaviorally equivalent actions performed by different robots, the model learns to ignore morphology-specific joint configurations in favor of abstract action intents.

Distribution Matching via Variational Encoders

Another approach uses generative modeling to map heterogeneous spaces into a shared distribution. Yin et al. in Cross Domain Robot Imitation with Invariant Representation utilize variational encoders to project diverse physical inputs into a common latent manifold, bridging the gap between different agent configurations without requiring tedious, human-labeled pairwise data. Similarly, Zhang et al. developed a plug-and-play framework in Align-Then-stEer: Adapting the Vision-Language Action Models through Unified Latent Guidance that uses a variational autoencoder to align a target robot's action distribution with a pre-trained model's latent space, allowing generalist models to interpret new hardware configurations easily.

Optimal Transport

When the appearance and kinematic gaps are particularly severe—such as translating first-person human videos to a robotic arm—distribution matching can be explicitly guided by geometric alignment. In EgoBridge: Domain Adaptation for Generalizable Imitation from Egocentric Human Data, Punamiya et al. employ Optimal Transport to align the joint distribution of observations and actions across domains. This anchors the alignment on functional actions rather than raw visual features, ensuring that behaviorally equivalent motions yield similar latent representations.

Learning from the Wild

Once actions are abstracted into a unified latent space, the boundary between different data domains dissolves. Robots can suddenly leverage unstructured, unlabeled data from humans or other robotic platforms.

Internet Videos as a Universal Interface

Instead of relying on expensive, annotated robotics datasets, world models can be trained directly on internet videos. Garrido et al. demonstrate in Learning Latent Action World Models in the Wild that action spaces derived from unlabeled internet videos can function as a universal interface for downstream robotic control. By training a minimal mapping layer, these general-purpose latent actions allow models to perform planning and navigation on specific robotic hardware without requiring large-scale labeled datasets for that specific robot.

This builds on earlier insights by Xie et al. in Learning Predictive Models from Observation and Interaction, which showed that mapping observations from different embodiments into a shared latent action space enables a robotic system to utilize unlabeled human video data to improve its own action-conditioned predictions and planning capabilities.

Grounding Language and Visual Plans

Unified latent spaces also allow high-level semantic commands to interface directly with low-level motor control. Lynch et al. show in Language Conditioned Imitation Learning Over Unstructured Data that mapping image-based goals and natural language instructions into a shared latent space allows language to act as a pointer to pre-learned motor behaviors, drastically reducing the need for expensive human annotations.

When an agent needs to execute entirely unseen tasks from language without any supervision, it can use video generative models to "imagine" a visual plan. In the RLZero framework developed by Sikchi et al. (RL Zero: Zero-Shot Language to Behaviors without any Supervision and RLZero: Direct Policy Inference from Language Without In-Domain Supervision), semantic similarity models align high-level features across different physical forms. This enables cross-embodiment imitation by matching state distributions in a shared latent space, bypassing the need for explicit retargeting software.

Decoding Intent into Execution

How does a robot turn an abstract latent action back into physical movement? The architecture typically operates as a two-stage pipeline:

  1. The Generalist Policy / World Model: Observes the current state and generates a trajectory of next actions within the unified latent space.
  2. The Embodiment-Specific Decoder: Translates the latent action vector into the exact joint torques or coordinate velocities required by the specific hardware.

This separation of concerns allows for powerful cross-platform generalization. For example, Liu et al. demonstrate in RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation that mapping disparate control signals into a physically interpretable vector space allows a foundation model to transfer learned physical principles across different hardware configurations, enabling knowledge gained from single-arm robots to be effectively applied to complex dual-arm systems.

Furthermore, grounding these representations within world models enables robust planning and one-shot imitation. As shown by Wang et al. in FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making, grounding semantic embeddings into a shared latent space allows a robot to translate the intent of a third-person demonstration into its own specific joint constraints, even when the demonstrator's body structure differs significantly. By stripping away embodiment-specific details, models can focus entirely on task-relevant physical transformations, a mechanism that Goswami et al. leverage in OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation to achieve one-shot imitation across different robotic platforms.

Even complex temporal and physical constraints can be managed this way. Zhao et al. demonstrate in SMAP: Self-supervised Motion Adaptation for Physically Plausible Humanoid Whole-body Control that a shared latent phase manifold can capture the underlying timing and rhythm of movements, allowing complex human motions to be translated into a format that respects the specific kinematic constraints of a humanoid robot. Finally, Zhang et al. show in Affordance-Driven Next-Best-View Planning for Robotic Grasping that integrating scene synthesis and task prediction within a shared latent space ensures that predicted outcomes from unvisited viewpoints remain grounded in consistent geometric features, allowing an agent to accurately evaluate potential actions from novel perspectives.

By shifting the paradigm from controlling joints to generating functional intent, unified latent action spaces provide the mathematical bridge necessary to make robot learning truly general, scalable, and cross-embodiment.

Key Foundational Work

Learning Predictive Models from Observation and Interaction (2020) — contributed 1 concept elements to this direction.
Cross Domain Robot Imitation with Invariant Representation (2021) — contributed 1 concept elements to this direction.
Language Conditioned Imitation Learning Over Unstructured Data (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_shared_latent_action_space_10278d9f,
  title = {shared latent action space},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/shared-latent-action-space/rdr_10278d9f330e46f2bcf3f9a6c0f277fa},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_shared_latent_action_space_10278d9f_v1,
  title = {shared latent action space},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/shared-latent-action-space/rdr_10278d9f330e46f2bcf3f9a6c0f277fa/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}

See something wrong? Report a mistake