TL;DR
JEPA4R is a novel world model that enables robots to plan and act directly within the 3D metric space of point clouds rather than 2D pixels. By utilizing a Joint-Embedding Predictive Architecture (JEPA) on 4D point clouds, it achieves state-of-the-art performance on manipulation benchmarks and demonstrates seamless zero-shot transfer from digital twins to real-world robotic arms.
What Problem Does This Paper Address?
Most current "world models"—AI systems that help robots predict the consequences of their actions—operate on 2D images. While 2D video is abundant, it creates two major headaches for robotics:
- Entangled Geometry: Images mix up the actual 3D structure of an object with its appearance and the camera's perspective. A robot needs to know exactly where an object is in centimeters, not just what pixel it occupies.
- Hidden Imaginations: Modern JEPA models predict abstract "embeddings" (mathematical vectors) rather than pixels. This makes them efficient, but it also makes them "black boxes." We can't easily see what the robot is "imagining" during its planning phase, which is a nightmare for debugging and safety.
The authors introduce JEPA4R to solve these issues by moving from 2D pixel-space to 4D point-space (3D coordinates over time).
Methods
JEPA4R follows the Joint-Embedding Predictive Architecture (JEPA) philosophy: instead of trying to reconstruct every pixel of a future frame (which is computationally expensive and often irrelevant), it predicts how the latent representation of a scene changes given a specific action.
The Architecture
- Encoder: A PointNet-based encoder takes a single-depth-camera point cloud (1,200 points with XYZ and RGB data) and compresses it into a 192-dimensional latent vector.
- Predictor: A six-block transformer predicts future latents based on current state and planned actions. It uses SIGReg (a regularization technique) to prevent the model from "collapsing" into trivial, useless solutions.
- Policy Head: A small network trained on top of the frozen world model latents to output actual motor commands.
- The 4D Decoder: This is a key innovation. The authors attached a post-hoc decoder (based on PAGE-4D) to the frozen world model. This allows them to turn those abstract 192-d vectors back into human-readable 3D point clouds. This "unhides" the robot's imagination, allowing researchers to verify that the robot's internal predictions match physical reality.
Datasets
The model was rigorously tested across several major robotics benchmarks:
- OGBench: Tasks involving moving, stacking, and cycling cubes (Cube-Single, Double, Triple) and complex scene interactions.
- CortexBench: Includes MetaWorld (5 distinct manipulation tasks like drawer opening and hammering) and Adroit Relocate (a high-dimensional task using a dexterous hand to move a ball).
- Real-World Deployment: The model was trained exclusively in a digital twin and then deployed zero-shot on a RealMan RM75-B robotic arm equipped with both a parallel gripper and a five-fingered dexterous hand.
Key Insights and Ideas
- 3D-Native Planning: By using point clouds, the metric geometry is explicit. The model doesn't have to "learn" depth; it's provided in the input. This makes the features significantly more robust for manipulation tasks.
- Decodability as a Diagnostic: The authors proved that even though the JEPA wasn't trained to reconstruct images, its latents naturally retain high-fidelity geometric information. The decoded "imagined" rollouts maintained a mean point error of less than 1.1 cm over 200 steps, which is remarkable for a non-reconstructive model.
- Sim-to-Real via Geometry: 3D geometry is more consistent between simulation and reality than 2D textures or lighting. This allowed JEPA4R to work on a real robot arm immediately after training in a "digital twin" without any fine-tuning on real-world data.
Key Results
JEPA4R consistently outperformed 2D-based JEPA models and other state-of-the-art representations:
- OGBench Cube-Single: Achieved a 78.4% success rate, nearly doubling the performance of the best 2D JEPA (PLDM at 46.8%).
- MetaWorld: Ranked first with a 98.4% mean success rate, outperforming the popular R3M representation (96.0%).
- Adroit Relocate: On this difficult dexterous hand task, it reached 88.0% success, far exceeding the runner-up's 65.3%.
- Zero-Shot Success: In real-world tests, the robot successfully performed picking and pushing tasks with a gripper and pulled various objects (bottles, bananas, carrots) from holders using a dexterous hand, all without seeing a single frame of real-world training data.
Limitations and Future Work
While JEPA4R is a significant step forward, the authors note a few areas for growth:
- Data Scarcity: High-quality, diverse 3D robotic data is still much harder to find than 2D internet video. Scaling this to a "foundation model" level will require massive 3D datasets.
- Single Viewpoint: The current model relies on a single fixed depth camera. Future iterations could benefit from multi-view inputs to handle occlusions better.
- Generalization: While it transfers well to a real arm, creating a truly "general" world model that works across vastly different robot morphologies remains an open challenge.
In summary, JEPA4R suggests that the future of embodied AI might not be in pixels, but in the explicit, measurable world of 3D points.
Summary updated