TL;DR
ReLAP is a new framework for teaching robots how to act by watching unlabeled videos. By forcing a model to "imagine" a sequence of future states in a compressed feature space (JEPA), it learns temporally coherent latent actions that translate remarkably well to real-world robot control, achieving state-of-the-art performance on manipulation benchmarks with significantly fewer parameters than previous models.
What Problem Does This Paper Address?
The "Holy Grail" of robotics is learning from the vast amount of video data available on the internet. However, these videos lack "action labels"—we see a hand move a cup, but we don't know the exact joint torques or velocities involved.
To solve this, researchers use Latent Action Models (LAMs), which infer "pseudo-actions" from visual changes. The problem is that most LAMs only look at pairwise transitions (Frame A to Frame B). In complex tasks like manipulation, actions are part of a sequence: reaching for a handle changes the state for the grasp, which in turn sets up the lift. If a model only learns one-step transitions, it fails to capture this temporal structure, leading to "drift" and incoherent behavior when the robot actually tries to execute a plan.
Methods
ReLAP (Recurrent Latent Action Pretraining) introduces a three-stage approach to bridge the gap between watching videos and executing actions.
- Stage 1: Latent Discovery in JEPA Space. Instead of predicting raw pixels (which is computationally expensive and noisy), ReLAP operates in a frozen V-JEPA 2.1 representation space. It uses an Inverse Dynamics Model (IDM) to infer a latent action from two frames and a Latent World Model (LWM) to predict the resulting feature map.
- The Innovation: Unlike standard models that predict one step at a time from ground-truth frames, ReLAP performs a recurrent rollout. It feeds its own predicted features back into the model for the next step. This allows errors at the end of a sequence to backpropagate and "supervise" the actions at the beginning, ensuring the whole sequence is temporally coherent.
- Stage 2: VLM Pretraining. The learned latent actions are transferred to a Vision-Language Model (VLM). The VLM learns to predict an entire sequence of these latent actions given just a single starting image and a text instruction.
- Stage 3: Robot Adaptation. Finally, the model is fine-tuned on a small set of labeled robot demonstrations. Using action flow matching, the abstract latent actions are mapped to real physical control signals.
Datasets
The authors pretrained ReLAP on a massive mixture of seven video datasets, totaling 11.1 million training windows. This mix includes both human and robot interactions:
- Robot-centric: Bridge-v2, DROID, AgiBot, Airoa-MOMA, InternData.
- Human-centric: EgoDex, Ego4D-v2.
Evaluation was conducted on the LIBERO manipulation benchmark (40 tasks), LIBERO-Plus (for zero-shot robustness), and the LARY benchmark for latent action quality.
Key Insights and Ideas
- Recurrence as Supervision: The "aha" moment is that you don't need complex mathematical constraints to make actions "smooth." By simply making the model's future predictions dependent on its past predictions, temporal coherence emerges naturally.
- JEPA is a "Safe" Sandbox: Operating in a frozen feature space (JEPA) acts as a powerful regularizer. It forces the model to focus on the physics and semantics of the scene rather than getting distracted by changing lights or textures.
- Efficiency Matters: ReLAP achieves top-tier results with only 1.06B trainable parameters. For comparison, many competing models use 7B+ parameter backbones. This suggests that the way we structure pretraining is more important than just scaling up the model size.
Key Results
- LIBERO Benchmark: ReLAP achieved a 97.9% average success rate, outperforming or matching state-of-the-art models like RotVLA and ALAM.
- Zero-Shot Robustness: On LIBERO-Plus, which perturbs things like object layout and lighting, ReLAP maintained a 70.0% success rate without any additional training on those perturbations. It was particularly strong against changes in object layout (90.1%).
- LARY Benchmark: ReLAP ranked second overall in both regression error and classification accuracy, proving that its "latent actions" are highly representative of real physical movements.
Limitations and Future Work
- Temporal Asymmetry: Because of the recurrent rollout, earlier actions in a sequence receive more gradient signal than later ones. This can lead to the model being very accurate at the start of a task but drifting toward the end of its 1.6-second prediction horizon.
- Multi-View Invariance: The current model is primarily trained on a single "main" camera view. While it uses a wrist camera during robot execution, the pretraining doesn't explicitly force the model to understand that different camera angles are looking at the same physical action.
- Horizon Scaling: The current rollout is limited to 4 steps (1.6 seconds). Scaling this to much longer horizons for complex, multi-stage tasks remains an open challenge.
Summary updated