SALA: Semantically Aligned Latent Actions Enable Dual-Axis Scaling of Embodied Foundation Models

Jiachun Jin, Zetong Zhou, Haonan Wang, Xiao Yang, Jun Zhu, Zhijie Deng

TL;DR

SALA (Semantically Aligned Latent Actions) allows robots to learn manipulation skills from massive amounts of action-free human video by using language as a bridge. By aligning human and robot behaviors in a shared, action-centric latent space, SALA enables "dual-axis scaling": pre-training on general human videos to reduce prediction error and post-training on task-specific human demonstrations to significantly boost real-world success rates.

What Problem Does This Paper Address?

Training "Embodied Foundation Models" (like Vision-Language-Action models) requires a staggering amount of robot interaction data, which is notoriously expensive and slow to collect via teleoperation. While egocentric human videos are abundant, they suffer from a massive embodiment gap: humans and robots have different joints, move differently, and operate in different environments.

Existing Latent Action Models (LAMs) try to bridge this gap by learning a shared representation of visual transitions. However, because they typically rely on video reconstruction (predicting the next frame), they often "cheat" by clustering actions based on visual appearance (the scene) rather than the actual underlying motion. This makes it difficult for a robot to recognize that a human "grasping a cup" in a kitchen is doing the same fundamental action as a robot "grasping a cup" in a lab.

Methods

The core of SALA is a Latent Action Tokenizer that organizes actions by their meaning (semantics) rather than their looks.

  1. Language as the Bridge: Even if human and robot videos are unpaired, they can be described with the same language (e.g., "grasp the purple toy"). SALA uses a frozen text encoder to measure the semantic similarity between these descriptions.
  2. Robot-Anchored Alignment: Robot latent actions are grounded in actual control data (reconstructing real robot actions). These act as "anchors." Human latent actions are then guided toward these semantically related robot anchors using a soft neighborhood structure.
  3. SoftVQ Quantization: Instead of using "hard" discrete tokens (which can be brittle), SALA uses SoftVQ, representing tokens as categorical distributions. This allows the model to express graded differences between similar actions.
  4. Dual-Axis Scaling: Once the tokenizer is trained, it can label any video with "latent action tokens." This allows the policy to scale in two ways:
  5. Pre-training: Learning from thousands of hours of general human video to build a broad understanding of manipulation.
  6. Post-training: Using task-specific human demonstrations (which are easy to record) to fine-tune the robot for a new task without needing more robot data.
Overview of SALA: Using language to bridge the embodiment gap and enable scaling with human video.

Datasets

  • Simulation: Evaluated on RoboTwin 2.0, a bimanual manipulation benchmark. The authors used 27,000 trajectories from an ARX-X5 robot as "human" data and 2,500 trajectories from an Aloha-AgileX robot as the target.
  • Real-World:
    • Human Video ($D_H$): 2,678 equivalent hours of egocentric human video.
  • Robot Data ($D_R$): 888 hours of teleoperation data from a Sharpa North humanoid robot (63-DoF, including two 22-DoF dexterous hands).
  • Downstream Tasks: Gear assembly, bimanual pouring, flower insertion, and spoon scooping.

Key Insights and Ideas

  • Semantics Over Scenes: Without alignment, latent spaces naturally cluster by background/lighting. SALA forces the model to ignore the "where" and focus on the "what." As shown in Figure 5, language alignment shifts the latent variance from being 14% explained by the scene to only 4%, while increasing the variance explained by the action from 41% to 67%.
  • One-Way Alignment: The authors use a "stop-gradient" on robot latents during alignment. This ensures the human data moves toward the robot's control-grounded space, rather than pulling the robot's space toward ungrounded human motions.
  • Human Demos as Cheap Supervision: Because the latent space is shared, a human performing a task in front of a camera provides a valid supervision signal for the robot's action expert. This makes "post-training" for new tasks much more scalable than traditional teleoperation.
Latent space visualization showing how language alignment reorganizes actions by semantics rather than scene.

Key Results

  • Simulation Superiority: On RoboTwin 2.0, SALA achieved a 64.4% average success rate, outperforming all tested LAM and World-Action Model (WAM) baselines (the best baseline hit 51.2%).
  • Real-World Scaling:
    • Pre-training: Increasing human video pre-training from 0 to 2,678 hours consistently reduced robot action prediction error (a 4.10% reduction compared to a vanilla VLA).
  • Post-training: On a real 63-DoF robot, adding task-specific human demonstrations raised the average success rate from 53.75% to 72.5%.
  • Robustness: SALA proved particularly effective in domain-randomized scenes, where visual-only models often fail due to changing backgrounds.
Scaling results showing reduced error with more pre-training and higher success rates with more human demonstrations.

Limitations and Future Work

  • Annotation Requirement: The tokenizer training still requires atomic-action descriptions (e.g., "grasp," "release"). While the final policy doesn't need them, generating these for massive datasets is a bottleneck.
  • Evaluation Scale: The scaling analysis currently uses offline validation error for the largest pre-training budgets; the authors note that how this translates to real-world success at even larger scales (e.g., 10k+ hours) remains an open question.
  • Complex Dynamics: While effective for pick-and-place style tasks, the paper doesn't deeply explore tasks with highly complex contact dynamics or high-speed motions where the human-robot gap might be even wider.

Summary updated

Abstract

Embodied foundation models hinge on diverse robot interaction data, which are costly to collect. Egocentric human videos offer a complementary source of experience, and latent action models (LAMs) provide a popular means of leveraging these videos for policy supervision. Compared with world-action models (WAMs), LAMs can offer greater training efficiency and potentially lower inference costs by avoiding explicit pixel prediction during policy learning and execution. However, most existing LAMs learn latent actions solely through video reconstruction, which can cause their representations to cluster by visual appearance rather than by the underlying actions. As a result, they struggle to align human actions with their robot counterparts. To bridge the gap, we introduce SALA, which emphasizes establishing the semantic correspondence for latent actions. Without requiring paired data, we use the language descriptions of unpaired human and robot clips to estimate semantic similarities and supervise cross-embodiment correspondence. Specifically, we translate these similarities into soft neighborhood structures that guide human latent actions toward semantically related robot latents, while robot action reconstruction grounds the latent space in control. The resulting shared latent action space provides a unified supervision signal for training an embodied foundation model on both human and robot videos. Notably, latent action supervision enables scaling along two axes: (1) pre-training on increasing amounts of general human video consistently reduces robot action prediction error on downstream tasks, and (2) post-training on increasing amounts of task-specific human video substantially improves real-world robot task success rates.
BibTeX
@misc{jin2026sala,
  author = {Jiachun Jin and Zetong Zhou and Haonan Wang and Xiao Yang and Jun Zhu and Zhijie Deng},
  title = {{SALA: Semantically Aligned Latent Actions Enable Dual-Axis Scaling of Embodied Foundation Models}},
  year = {2026},
  eprint = {2611.17110},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17110},
}
Figure