TL;DR
EgoExo-WM overcomes the severe data scarcity limiting egocentric (first-person) world models by unlocking the vast world of internet-scale exocentric (third-person) video. By using 3D human motion as a physical and kinematic bridge, the framework translates third-person video into action-aligned first-person experiences, significantly improving future prediction and downstream goal-conditioned planning for embodied agents.
What Problem Does This Paper Address?
Building "world models"—predictive models that allow an AI or robotic agent to anticipate how an environment changes in response to actions—is a cornerstone of embodied intelligence. For agents interacting with humans or operating from a first-person perspective (like AR glasses or humanoid robots), an egocentric world model is ideal.
However, training these models faces a dual bottleneck:
- Data Scarcity: Egocentric video is notoriously difficult and expensive to collect at scale, requiring subjects to wear specialized camera rigs across diverse environments. In contrast, standard third-person (exocentric) video is abundant across the internet (e.g., YouTube cooking tutorials).
- Partial Observability: First-person views naturally hide the actor's own body and frequently occlude the hands during object manipulation, making it incredibly hard for a model to learn the direct relationship between full-body kinematics and visual outcomes.
The authors tackle a fundamental question: How can we leverage abundant, full-body third-person internet videos to train first-person predictive world models?
Methods
The core innovation of EgoExo-WM is using 3D human motion as a unifying bridge between third-person observation, video synthesis, and world model learning.
The framework consists of two main stages:
1. Exo-to-Ego Data Generation (EgoX-Body)
To turn third-person video into first-person training data, the authors introduce EgoX-Body.
- Kinematic Extraction: They extract full-body 3D poses from exocentric video using a pose estimator (
SAM-Body4D) and lift the scene geometry. - Dual-Prior Conditioning: They condition a video diffusion model by overlaying 3D human skeletons onto the exocentric frames and projecting estimated 3D hand positions into a synthesized egocentric viewpoint. This ensures the generated first-person video is tightly aligned with actual physical actions rather than just being visually plausible.
2. Action-Conditioned World Model Training
Once the data is converted into a paired egocentric format (visual observations + 3D human actions), the world model is trained:
- Latent Space Prediction: Instead of expensive pixel-level generation, the model operates within a compact visual latent space using a pretrained
DINOv3-Lencoder. Given a history of past latents and a 3D human motion action (parameterized as a 69-dimensional SMPL joint vector), it predicts the next visual latent. - Wrist-Position Consistency Loss: To prevent the model from drifting or ignoring the agent's posture, they attach a lightweight, frozen transformer head that predicts a 2D wrist heatmap from the predicted latent space, penalizing deviations from the actual hand trajectories.
Datasets
- Egocentric Training Data:
Nymeria(~200 hours of synchronized first-person video and full-body motion capture). - Exocentric Source Data (Converted): ~10 hours of clips sampled from
HowTo100M(instructional),CrossTask(task-oriented), and100 Days of Hands(hand-object interactions). - Evaluation Benchmarks: Evaluated on diverse, held-out paired datasets:
Home Action Genome (HOMAGE),LEMMA, and the Bike and Cooking subsets ofEgo-Exo4D.
Key Insights and Ideas
- 3D Pose as a Universal Action Space: Rather than relying on abstract latent actions or coarse text descriptions, 3D human kinematics provide an interpretable, physically grounded action space common to both third-person and first-person perspectives.
- Purely Visual Translation is Not Enough: Standard image-to-image or view-synthesis models often hallucinate or swap hands because they lack physical constraints. Explicitly rendering hand and body skeletons into the diffusion process (as done in EgoX-Body) is essential for action-conditioned consistency.
- Geometry-Grounded Latents: Forcing the latent space of the world model to explicitly decode hand positions (via the wrist consistency loss) ensures that semantic features remain anchored to the agent's physical embodiment.
Key Results
Direct Evaluation of World Model Accuracy
EgoExo-WM consistently outperformed state-of-the-art egocentric video prediction baselines (PEVA and EgoControl) across all evaluation datasets. On HOMAGE and LEMMA, EgoExo-WM reduced the average visual embedding error ($L_2$ distance) by more than half compared to the strongest baseline, proving that the injected, converted exocentric data successfully expanded the model's understanding of diverse environments and actions.
Downstream Model-Predictive Control (MPC) Planning
The authors tested the world model in a goal-conditioned planning setting: given a current first-person view and a visual goal image, the model had to select the best sequence of 3D human motions to achieve that goal.
Using EgoExo-WM to rank candidate trajectories generated by an action proposal model (UniEgoMotion) yielded the lowest Mean Per Joint Position Error (MPJPE) across the board, demonstrating that a more accurate world model directly translates to superior robotic or agent planning.
Limitations and Future Work
- Temporal Scale: The framework currently operates on short horizons (49-frame clips, roughly a 2-second horizon). Scaling to long-horizon planning where prediction errors compound remains an open challenge.
- Fine-Grained Manipulation: While full-body kinematics are well-captured, the model still struggles with highly complex human-object interactions involving severe occlusions, precise contact dynamics, or very small objects.
- Synthesis Artifacts: Occasionally, the exo-to-ego conversion module generates degenerated (mostly black or white) frames when faced with extreme viewpoint mismatches.
Summary updated