ReLAP: Latent Action Pretraining through Recurrent Video Prediction in JEPA Space

R. Gnana Praveen, Ali Rouzbayani, Zhiying Jiang, Bidhan RoyarXiv 2026 · arXiv

TL;DR

ReLAP is a new framework for teaching robots how to act by watching unlabeled videos. By forcing a model to "imagine" a sequence of future states in a compressed feature space (JEPA), it learns temporally coherent latent actions that translate remarkably well to real-world robot control, achieving state-of-the-art performance on manipulation benchmarks with significantly fewer parameters than previous models.

What Problem Does This Paper Address?

The "Holy Grail" of robotics is learning from the vast amount of video data available on the internet. However, these videos lack "action labels"—we see a hand move a cup, but we don't know the exact joint torques or velocities involved.

To solve this, researchers use Latent Action Models (LAMs), which infer "pseudo-actions" from visual changes. The problem is that most LAMs only look at pairwise transitions (Frame A to Frame B). In complex tasks like manipulation, actions are part of a sequence: reaching for a handle changes the state for the grasp, which in turn sets up the lift. If a model only learns one-step transitions, it fails to capture this temporal structure, leading to "drift" and incoherent behavior when the robot actually tries to execute a plan.

Methods

ReLAP (Recurrent Latent Action Pretraining) introduces a three-stage approach to bridge the gap between watching videos and executing actions.

  1. Stage 1: Latent Discovery in JEPA Space. Instead of predicting raw pixels (which is computationally expensive and noisy), ReLAP operates in a frozen V-JEPA 2.1 representation space. It uses an Inverse Dynamics Model (IDM) to infer a latent action from two frames and a Latent World Model (LWM) to predict the resulting feature map.
  2. The Innovation: Unlike standard models that predict one step at a time from ground-truth frames, ReLAP performs a recurrent rollout. It feeds its own predicted features back into the model for the next step. This allows errors at the end of a sequence to backpropagate and "supervise" the actions at the beginning, ensuring the whole sequence is temporally coherent.
Comparison between conventional latent action models and ReLAP's recurrent approach.
  1. Stage 2: VLM Pretraining. The learned latent actions are transferred to a Vision-Language Model (VLM). The VLM learns to predict an entire sequence of these latent actions given just a single starting image and a text instruction.
  2. Stage 3: Robot Adaptation. Finally, the model is fine-tuned on a small set of labeled robot demonstrations. Using action flow matching, the abstract latent actions are mapped to real physical control signals.
The three stages of the ReLAP framework: discovery, VLM pretraining, and robot adaptation.

Datasets

The authors pretrained ReLAP on a massive mixture of seven video datasets, totaling 11.1 million training windows. This mix includes both human and robot interactions:

  • Robot-centric: Bridge-v2, DROID, AgiBot, Airoa-MOMA, InternData.
  • Human-centric: EgoDex, Ego4D-v2.

Evaluation was conducted on the LIBERO manipulation benchmark (40 tasks), LIBERO-Plus (for zero-shot robustness), and the LARY benchmark for latent action quality.

The distribution of the 11.1M training windows across various datasets.

Key Insights and Ideas

  • Recurrence as Supervision: The "aha" moment is that you don't need complex mathematical constraints to make actions "smooth." By simply making the model's future predictions dependent on its past predictions, temporal coherence emerges naturally.
  • JEPA is a "Safe" Sandbox: Operating in a frozen feature space (JEPA) acts as a powerful regularizer. It forces the model to focus on the physics and semantics of the scene rather than getting distracted by changing lights or textures.
  • Efficiency Matters: ReLAP achieves top-tier results with only 1.06B trainable parameters. For comparison, many competing models use 7B+ parameter backbones. This suggests that the way we structure pretraining is more important than just scaling up the model size.

Key Results

  • LIBERO Benchmark: ReLAP achieved a 97.9% average success rate, outperforming or matching state-of-the-art models like RotVLA and ALAM.
  • Zero-Shot Robustness: On LIBERO-Plus, which perturbs things like object layout and lighting, ReLAP maintained a 70.0% success rate without any additional training on those perturbations. It was particularly strong against changes in object layout (90.1%).
  • LARY Benchmark: ReLAP ranked second overall in both regression error and classification accuracy, proving that its "latent actions" are highly representative of real physical movements.

Limitations and Future Work

  • Temporal Asymmetry: Because of the recurrent rollout, earlier actions in a sequence receive more gradient signal than later ones. This can lead to the model being very accurate at the start of a task but drifting toward the end of its 1.6-second prediction horizon.
  • Multi-View Invariance: The current model is primarily trained on a single "main" camera view. While it uses a wrist camera during robot execution, the pretraining doesn't explicitly force the model to understand that different camera angles are looking at the same physical action.
  • Horizon Scaling: The current rollout is limited to 4 steps (1.6 seconds). Scaling this to much longer horizons for complex, multi-stage tasks remains an open challenge.

Summary updated

Abstract

Videos provide a rich source of physical interaction for learning robot behavior without action annotations, yet effective latent actions must capture not only local transitions but also how those transitions evolve over time. We introduce ReLAP, a latent-action pretraining framework that couples local transition learning with recurrent multi-step prediction in a frozen JEPA representation space. A shared inverse dynamics model infers compact continuous latent actions from adjacent observations, while a latent world model recurrently predicts the future JEPA features. Because each predicted feature becomes the context for the next transition, errors at later steps backpropagate through the rollout to earlier latent actions, encouraging temporally coherent dynamics beyond pairwise reconstruction. We transfer this representation to a vision-language model using two complementary objectives: latent distillation from the frozen inverse teacher and recurrent future-feature prediction through the frozen latent world model. The resulting policy is then adapted to robot demonstrations with action flow matching. At deployment, the explicit latent-action and world-model branches are removed, and the fine-tuned VLM predicts physical actions directly, reducing inference complexity. With only 11M pretraining samples and 1.06B trainable parameters, ReLAP achieves 97.9% on standard LIBERO and 70.0% over zero-shot LIBERO-Plus perturbations. On the LARY latent-action benchmark, ReLAP ranks second in both mean regression error and mean classification accuracy among the compared methods. These results demonstrate that recurrent action-free latent action pretraining can learn compact, temporally structured, and transferable representations for downstream robot control.
BibTeX
@misc{praveen2026relap,
  author = {R. Gnana Praveen and Ali Rouzbayani and Zhiying Jiang and Bidhan Roy},
  title = {{ReLAP: Latent Action Pretraining through Recurrent Video Prediction in JEPA Space}},
  year = {2026},
  eprint = {2611.22077},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22077},
}
Figure