JEPA4R: Embodied JEPA World Models with 4D Point Planning

Jiarui Dong, Zhiheng Liang, Yanjun Li, Zeyu Zhang, Heejeong Nam, Haoyu Chen, Hao TangarXiv 2026 · arXiv

TL;DR

JEPA4R is a novel world model that enables robots to plan and act directly within the 3D metric space of point clouds rather than 2D pixels. By utilizing a Joint-Embedding Predictive Architecture (JEPA) on 4D point clouds, it achieves state-of-the-art performance on manipulation benchmarks and demonstrates seamless zero-shot transfer from digital twins to real-world robotic arms.

What Problem Does This Paper Address?

Most current "world models"—AI systems that help robots predict the consequences of their actions—operate on 2D images. While 2D video is abundant, it creates two major headaches for robotics:

  1. Entangled Geometry: Images mix up the actual 3D structure of an object with its appearance and the camera's perspective. A robot needs to know exactly where an object is in centimeters, not just what pixel it occupies.
  2. Hidden Imaginations: Modern JEPA models predict abstract "embeddings" (mathematical vectors) rather than pixels. This makes them efficient, but it also makes them "black boxes." We can't easily see what the robot is "imagining" during its planning phase, which is a nightmare for debugging and safety.

The authors introduce JEPA4R to solve these issues by moving from 2D pixel-space to 4D point-space (3D coordinates over time).

Methods

JEPA4R follows the Joint-Embedding Predictive Architecture (JEPA) philosophy: instead of trying to reconstruct every pixel of a future frame (which is computationally expensive and often irrelevant), it predicts how the latent representation of a scene changes given a specific action.

The Architecture

  • Encoder: A PointNet-based encoder takes a single-depth-camera point cloud (1,200 points with XYZ and RGB data) and compresses it into a 192-dimensional latent vector.
  • Predictor: A six-block transformer predicts future latents based on current state and planned actions. It uses SIGReg (a regularization technique) to prevent the model from "collapsing" into trivial, useless solutions.
  • Policy Head: A small network trained on top of the frozen world model latents to output actual motor commands.
  • The 4D Decoder: This is a key innovation. The authors attached a post-hoc decoder (based on PAGE-4D) to the frozen world model. This allows them to turn those abstract 192-d vectors back into human-readable 3D point clouds. This "unhides" the robot's imagination, allowing researchers to verify that the robot's internal predictions match physical reality.

Datasets

The model was rigorously tested across several major robotics benchmarks:

  • OGBench: Tasks involving moving, stacking, and cycling cubes (Cube-Single, Double, Triple) and complex scene interactions.
  • CortexBench: Includes MetaWorld (5 distinct manipulation tasks like drawer opening and hammering) and Adroit Relocate (a high-dimensional task using a dexterous hand to move a ball).
  • Real-World Deployment: The model was trained exclusively in a digital twin and then deployed zero-shot on a RealMan RM75-B robotic arm equipped with both a parallel gripper and a five-fingered dexterous hand.

Key Insights and Ideas

  • 3D-Native Planning: By using point clouds, the metric geometry is explicit. The model doesn't have to "learn" depth; it's provided in the input. This makes the features significantly more robust for manipulation tasks.
  • Decodability as a Diagnostic: The authors proved that even though the JEPA wasn't trained to reconstruct images, its latents naturally retain high-fidelity geometric information. The decoded "imagined" rollouts maintained a mean point error of less than 1.1 cm over 200 steps, which is remarkable for a non-reconstructive model.
  • Sim-to-Real via Geometry: 3D geometry is more consistent between simulation and reality than 2D textures or lighting. This allowed JEPA4R to work on a real robot arm immediately after training in a "digital twin" without any fine-tuning on real-world data.

Key Results

JEPA4R consistently outperformed 2D-based JEPA models and other state-of-the-art representations:

  • OGBench Cube-Single: Achieved a 78.4% success rate, nearly doubling the performance of the best 2D JEPA (PLDM at 46.8%).
  • MetaWorld: Ranked first with a 98.4% mean success rate, outperforming the popular R3M representation (96.0%).
  • Adroit Relocate: On this difficult dexterous hand task, it reached 88.0% success, far exceeding the runner-up's 65.3%.
  • Zero-Shot Success: In real-world tests, the robot successfully performed picking and pushing tasks with a gripper and pulled various objects (bottles, bananas, carrots) from holders using a dexterous hand, all without seeing a single frame of real-world training data.

Limitations and Future Work

While JEPA4R is a significant step forward, the authors note a few areas for growth:

  • Data Scarcity: High-quality, diverse 3D robotic data is still much harder to find than 2D internet video. Scaling this to a "foundation model" level will require massive 3D datasets.
  • Single Viewpoint: The current model relies on a single fixed depth camera. Future iterations could benefit from multi-view inputs to handle occlusions better.
  • Generalization: While it transfers well to a real arm, creating a truly "general" world model that works across vastly different robot morphologies remains an open challenge.

In summary, JEPA4R suggests that the future of embodied AI might not be in pixels, but in the explicit, measurable world of 3D points.

Summary updated

Abstract

Joint-embedding predictive architecture (JEPA) world models predict future embeddings instead of pixels and plan in latent space. Two challenges remain: 2D images leave the metric geometry that manipulation acts on implicit and entangled with appearance, and what these models imagine stays hidden, as they are mostly judged in simulation by short-horizon latent planning. We present JEPA4R, a JEPA world model that observes single-depth-camera point clouds, predicts and plans in their latent space, and acts through a policy head on its frozen latents; we call this 4D point planning. A decoder on the frozen world model shows what it imagines as 4D point clouds, with a mean point error within 1.1 cm over 200 steps. Under the policy-learning protocols of OGBench and CortexBench, JEPA4R leads pixel JEPA models on OGBench Cube-Single, Cube-Double, Cube-Triple and Scene, and ranks first on MetaWorld and Adroit Relocate with 98.4% and 88.0%, above the runner-up, R3M, at 96.0% and 65.3%. Trained only in a digital twin, JEPA4R and its policy heads run zero-shot on the real arm with both a gripper and a dexterous hand.
BibTeX
@misc{dong2026jepa,
  author = {Jiarui Dong and Zhiheng Liang and Yanjun Li and Zeyu Zhang and Heejeong Nam and Haoyu Chen and Hao Tang},
  title = {{JEPA4R: Embodied JEPA World Models with 4D Point Planning}},
  year = {2026},
  eprint = {2611.19771},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19771},
}
Figure