TL;DR
ELA (End-to-End Latent Action) is a framework that enables robots to learn complex manipulation tasks from abundant, action-free video demonstrations (like human videos) combined with just a few labeled robot examples. By jointly optimizing a latent action space for both policy predictability and robot execution, ELA significantly outperforms existing imitation learning methods in data-scarce regimes.
What Problem Does This Paper Address?
Imitation learning—teaching robots by showing them what to do—is powerful but bottlenecked by data. Collecting "action-labeled" data (where every video frame is paired with precise robot joint commands) is expensive and slow. In contrast, "action-free" video data, such as humans performing tasks or unlabeled robot footage, is abundant.
Recent research has tried to use "latent actions"—abstract representations of movement inferred from video—to bridge this gap. However, these methods usually follow a two-stage process: first, they learn a latent space from video, and then they try to train a robot policy on top of it. The authors of ELA argue this is suboptimal. If the latent space isn't explicitly designed for the robot's specific control needs, the resulting "actions" might be impossible for a policy to predict accurately (modelability) or impossible for a decoder to translate back into smooth robot motions (decodability).
Methods
ELA introduces an end-to-end framework that jointly trains three core components to ensure the latent action space is useful for downstream control:
- Latent Action Model (LAM): This consists of an Inverse Dynamics Model (which looks at two frames and guesses the "latent action" that happened between them) and a Forward Dynamics Model (which uses that latent action to predict the next frame, ensuring the representation is grounded in physical reality).
- Latent Action Policy ($\pi$): Instead of predicting raw robot commands, this policy learns to predict the latent actions from a single observation.
- Action Decoder ($\psi$): A lightweight (4M parameter) module that maps these latent actions into continuous, executable robot commands.
The "secret sauce" is Joint Training with Representation Alignment. To prevent the models from finding "shortcuts" that lead to latent action collapse, the authors alternate between updating the LAM and the Policy. They use a frozen perceptual encoder (DINOv3) as a reference to keep the latent features aligned with semantically meaningful visual structures.
Datasets
The authors evaluated ELA across two main benchmarks:
- LIBERO-10: A challenging simulation benchmark consisting of 10 long-horizon tabletop manipulation tasks (e.g., putting a book on a caddy, heating a mug in a microwave).
- Real-World Cross-Embodiment Suite: A set of 5 tasks (like pouring from a container or opening an oven) performed by a Franka Panda robot. Crucially, the training data included human video demonstrations to test the model's ability to learn from different "embodiments."
Key Insights and Ideas
- The Modelability-Decodability Trade-off: A good latent action isn't just a compressed representation of a video; it must be something a neural network can easily predict from a single image and something a motor controller can actually execute. ELA is the first to optimize for both simultaneously.
- End-to-End is Better than Two-Stage: By allowing the policy's needs to shape the latent action space during training, the system avoids the "representation gap" that plagues prior methods.
- Lightweight Integration: ELA adds only a 4M-parameter overhead and is compatible with existing policy architectures like Diffusion Policies. This makes it a "plug-and-play" upgrade for current imitation learning pipelines.
- Representation Alignment as a Stabilizer: Using a frozen vision foundation model (DINOv3) to guide the latent space prevents the joint optimization from diverging or collapsing into trivial solutions.
Key Results
ELA consistently outperformed state-of-the-art baselines, especially when labeled data was extremely limited:
- Simulation (LIBERO-10): With only 5 labeled demonstrations, ELA improved the success rate of a Unified World Model (UWM) from 19% to 59%. It also outperformed specialized methods like ATM and AMPLIFY that require complex preprocessing like point tracking.
- Real-World Success: In cross-embodiment tasks using human videos, ELA achieved a 70% average success rate, nearly doubling the performance of the standard Diffusion Policy (48%) and dwarfing other latent action methods like LAPA (26%) and Villa-X (27%).
As shown in Figure 5(c) above, ELA achieves the best balance of low decodability error and low modelability error, placing it in the "better" lower-left region of the graph compared to all other baselines.
Limitations and Future Work
While ELA shows great promise, the authors note several areas for growth:
- Scale: The method hasn't yet been tested on massive, internet-scale cross-embodiment datasets.
- 3D Priors: ELA operates on raw RGB pixels. Incorporating depth information or 3D foundation models could improve spatial reasoning.
- Dexterity: The current evaluation focused on general manipulation; highly precise or dexterous tasks (like threading a needle) remain a challenge for future iterations.
For more details, check out the project website.
Summary updated