ELA: End-to-End Latent Action Policy Learning from Video Demonstrations

Cheng-Chun Hsu, Peter StonearXiv 2026 · arXiv

TL;DR

ELA (End-to-End Latent Action) is a framework that enables robots to learn complex manipulation tasks from abundant, action-free video demonstrations (like human videos) combined with just a few labeled robot examples. By jointly optimizing a latent action space for both policy predictability and robot execution, ELA significantly outperforms existing imitation learning methods in data-scarce regimes.

What Problem Does This Paper Address?

Imitation learning—teaching robots by showing them what to do—is powerful but bottlenecked by data. Collecting "action-labeled" data (where every video frame is paired with precise robot joint commands) is expensive and slow. In contrast, "action-free" video data, such as humans performing tasks or unlabeled robot footage, is abundant.

Recent research has tried to use "latent actions"—abstract representations of movement inferred from video—to bridge this gap. However, these methods usually follow a two-stage process: first, they learn a latent space from video, and then they try to train a robot policy on top of it. The authors of ELA argue this is suboptimal. If the latent space isn't explicitly designed for the robot's specific control needs, the resulting "actions" might be impossible for a policy to predict accurately (modelability) or impossible for a decoder to translate back into smooth robot motions (decodability).

Methods

ELA introduces an end-to-end framework that jointly trains three core components to ensure the latent action space is useful for downstream control:

  1. Latent Action Model (LAM): This consists of an Inverse Dynamics Model (which looks at two frames and guesses the "latent action" that happened between them) and a Forward Dynamics Model (which uses that latent action to predict the next frame, ensuring the representation is grounded in physical reality).
  2. Latent Action Policy ($\pi$): Instead of predicting raw robot commands, this policy learns to predict the latent actions from a single observation.
  3. Action Decoder ($\psi$): A lightweight (4M parameter) module that maps these latent actions into continuous, executable robot commands.
Figure 1: ELA Overview. Our framework extends existing behavior cloning methods, e.g. diffusion policies, with latent actions jointly optimized for policy learning and action decoding. The approach enables training on mixtures of action-labeled and video demonstrations with little overhead, requir- ing only a lightweight action decoder.

The "secret sauce" is Joint Training with Representation Alignment. To prevent the models from finding "shortcuts" that lead to latent action collapse, the authors alternate between updating the LAM and the Policy. They use a frozen perceptual encoder (DINOv3) as a reference to keep the latent features aligned with semantically meaningful visual structures.

Detailed Training and Deployment Flow

Datasets

The authors evaluated ELA across two main benchmarks:

  • LIBERO-10: A challenging simulation benchmark consisting of 10 long-horizon tabletop manipulation tasks (e.g., putting a book on a caddy, heating a mug in a microwave).
  • Real-World Cross-Embodiment Suite: A set of 5 tasks (like pouring from a container or opening an oven) performed by a Franka Panda robot. Crucially, the training data included human video demonstrations to test the model's ability to learn from different "embodiments."

Key Insights and Ideas

  • The Modelability-Decodability Trade-off: A good latent action isn't just a compressed representation of a video; it must be something a neural network can easily predict from a single image and something a motor controller can actually execute. ELA is the first to optimize for both simultaneously.
  • End-to-End is Better than Two-Stage: By allowing the policy's needs to shape the latent action space during training, the system avoids the "representation gap" that plagues prior methods.
  • Lightweight Integration: ELA adds only a 4M-parameter overhead and is compatible with existing policy architectures like Diffusion Policies. This makes it a "plug-and-play" upgrade for current imitation learning pipelines.
  • Representation Alignment as a Stabilizer: Using a frozen vision foundation model (DINOv3) to guide the latent space prevents the joint optimization from diverging or collapsing into trivial solutions.

Key Results

ELA consistently outperformed state-of-the-art baselines, especially when labeled data was extremely limited:

  • Simulation (LIBERO-10): With only 5 labeled demonstrations, ELA improved the success rate of a Unified World Model (UWM) from 19% to 59%. It also outperformed specialized methods like ATM and AMPLIFY that require complex preprocessing like point tracking.
  • Real-World Success: In cross-embodiment tasks using human videos, ELA achieved a 70% average success rate, nearly doubling the performance of the standard Diffusion Policy (48%) and dwarfing other latent action methods like LAPA (26%) and Villa-X (27%).
Figure 5: Real-world Results. (a) Comparative analysis of success rates on the real-world bench- mark. (b) Ablation study of representation alignment across a subset of task suites. (c) Modelability versus Decodability for each latent action spaces.

As shown in Figure 5(c) above, ELA achieves the best balance of low decodability error and low modelability error, placing it in the "better" lower-left region of the graph compared to all other baselines.

Limitations and Future Work

While ELA shows great promise, the authors note several areas for growth:

  • Scale: The method hasn't yet been tested on massive, internet-scale cross-embodiment datasets.
  • 3D Priors: ELA operates on raw RGB pixels. Incorporating depth information or 3D foundation models could improve spatial reasoning.
  • Dexterity: The current evaluation focused on general manipulation; highly precise or dexterous tasks (like threading a needle) remain a challenge for future iterations.

For more details, check out the project website.

Summary updated

Abstract

Robot imitation learning is limited by the scarcity of action-labeled data, while action-free demonstrations such as videos are abundant. Recent latent-action methods learn action-relevant representations from observation transitions, but they do not optimize latent actions for downstream control, leading to latent representations that are difficult for policies to model and difficult to accurately decode into continuous robot actions. This paper introduces ELA, an end-to-end framework for learning from video demonstrations via latent actions jointly optimized for policy learning and action decoding. ELA jointly trains a latent action model, latent action policy, and action decoder, enabling policies to leverage abundant video data with only a handful of action-labeled robot demonstrations. Experiments on simulated and real-world manipulation tasks show that ELA consistently outperforms prior latent action baselines. Moreover, it integrates seamlessly with existing imitation learning methods through lightweight 4M-parameter action decoders, requiring no changes to policy architectures or training objectives. Additional materials can be found on: https://end2end-latent-action.github.io/
BibTeX
@misc{hsu2026ela,
  author = {Cheng-Chun Hsu and Peter Stone},
  title = {{ELA: End-to-End Latent Action Policy Learning from Video Demonstrations}},
  year = {2026},
  eprint = {2611.05571},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.05571},
}
Figure