EgoExo-WM: Unlocking Exo Video for Ego World Models

Danny Tran, Roberto Martín-Martín, Kristen GraumanarXiv 2026 · arXiv

TL;DR

EgoExo-WM overcomes the severe data scarcity limiting egocentric (first-person) world models by unlocking the vast world of internet-scale exocentric (third-person) video. By using 3D human motion as a physical and kinematic bridge, the framework translates third-person video into action-aligned first-person experiences, significantly improving future prediction and downstream goal-conditioned planning for embodied agents.

What Problem Does This Paper Address?

Building "world models"—predictive models that allow an AI or robotic agent to anticipate how an environment changes in response to actions—is a cornerstone of embodied intelligence. For agents interacting with humans or operating from a first-person perspective (like AR glasses or humanoid robots), an egocentric world model is ideal.

However, training these models faces a dual bottleneck:

  1. Data Scarcity: Egocentric video is notoriously difficult and expensive to collect at scale, requiring subjects to wear specialized camera rigs across diverse environments. In contrast, standard third-person (exocentric) video is abundant across the internet (e.g., YouTube cooking tutorials).
  2. Partial Observability: First-person views naturally hide the actor's own body and frequently occlude the hands during object manipulation, making it incredibly hard for a model to learn the direct relationship between full-body kinematics and visual outcomes.

The authors tackle a fundamental question: How can we leverage abundant, full-body third-person internet videos to train first-person predictive world models?

Methods

The core innovation of EgoExo-WM is using 3D human motion as a unifying bridge between third-person observation, video synthesis, and world model learning.

The framework consists of two main stages:

1. Exo-to-Ego Data Generation (EgoX-Body)

To turn third-person video into first-person training data, the authors introduce EgoX-Body.

  • Kinematic Extraction: They extract full-body 3D poses from exocentric video using a pose estimator (SAM-Body4D) and lift the scene geometry.
  • Dual-Prior Conditioning: They condition a video diffusion model by overlaying 3D human skeletons onto the exocentric frames and projecting estimated 3D hand positions into a synthesized egocentric viewpoint. This ensures the generated first-person video is tightly aligned with actual physical actions rather than just being visually plausible.

2. Action-Conditioned World Model Training

Once the data is converted into a paired egocentric format (visual observations + 3D human actions), the world model is trained:

  • Latent Space Prediction: Instead of expensive pixel-level generation, the model operates within a compact visual latent space using a pretrained DINOv3-L encoder. Given a history of past latents and a 3D human motion action (parameterized as a 69-dimensional SMPL joint vector), it predicts the next visual latent.
  • Wrist-Position Consistency Loss: To prevent the model from drifting or ignoring the agent's posture, they attach a lightweight, frozen transformer head that predicts a 2D wrist heatmap from the predicted latent space, penalizing deviations from the actual hand trajectories.
Figure 1: Overview. Egocentric video provides an embodied view but often hides the body and occludes the hands, while exocentric video often reveals full-body motion (a). EgoExo-WM uses recovered 3D human motion as a bridge for learning an egocentric world-model with exocentric video: it defines the action sequence and guides exo-to-ego synthesis into action- aligned egocentric observations (b). The learned world model enables goal-conditioned planning for applications such as robotics and augmented reality by selecting actions that best reach a target state (c).

Datasets

  • Egocentric Training Data: Nymeria (~200 hours of synchronized first-person video and full-body motion capture).
  • Exocentric Source Data (Converted): ~10 hours of clips sampled from HowTo100M (instructional), CrossTask (task-oriented), and 100 Days of Hands (hand-object interactions).
  • Evaluation Benchmarks: Evaluated on diverse, held-out paired datasets: Home Action Genome (HOMAGE), LEMMA, and the Bike and Cooking subsets of Ego-Exo4D.

Key Insights and Ideas

  • 3D Pose as a Universal Action Space: Rather than relying on abstract latent actions or coarse text descriptions, 3D human kinematics provide an interpretable, physically grounded action space common to both third-person and first-person perspectives.
  • Purely Visual Translation is Not Enough: Standard image-to-image or view-synthesis models often hallucinate or swap hands because they lack physical constraints. Explicitly rendering hand and body skeletons into the diffusion process (as done in EgoX-Body) is essential for action-conditioned consistency.
  • Geometry-Grounded Latents: Forcing the latent space of the world model to explicitly decode hand positions (via the wrist consistency loss) ensures that semantic features remain anchored to the agent's physical embodiment.

Key Results

Direct Evaluation of World Model Accuracy

EgoExo-WM consistently outperformed state-of-the-art egocentric video prediction baselines (PEVA and EgoControl) across all evaluation datasets. On HOMAGE and LEMMA, EgoExo-WM reduced the average visual embedding error ($L_2$ distance) by more than half compared to the strongest baseline, proving that the injected, converted exocentric data successfully expanded the model's understanding of diverse environments and actions.

Downstream Model-Predictive Control (MPC) Planning

The authors tested the world model in a goal-conditioned planning setting: given a current first-person view and a visual goal image, the model had to select the best sequence of 3D human motions to achieve that goal.

Using EgoExo-WM to rank candidate trajectories generated by an action proposal model (UniEgoMotion) yielded the lowest Mean Per Joint Position Error (MPJPE) across the board, demonstrating that a more accurate world model directly translates to superior robotic or agent planning.

Limitations and Future Work

  • Temporal Scale: The framework currently operates on short horizons (49-frame clips, roughly a 2-second horizon). Scaling to long-horizon planning where prediction errors compound remains an open challenge.
  • Fine-Grained Manipulation: While full-body kinematics are well-captured, the model still struggles with highly complex human-object interactions involving severe occlusions, precise contact dynamics, or very small objects.
  • Synthesis Artifacts: Occasionally, the exo-to-ego conversion module generates degenerated (mostly black or white) frames when faced with extreme viewpoint mismatches.

Summary updated

Abstract

Egocentric world models present a promising direction for enabling agents to predict and plan, but their performance is constrained by the limited availability of egocentric training data and its inherent partial observability of humans' physical actions. In contrast, exocentric video is abundant and reveals body poses well, but lacks direct alignment with an agent's action space -- and is not egocentric. We propose a method to bridge this gap by extracting structured body pose from exocentric video as a representation of action and transforming the exocentric video to egocentric video, informed by a human kinematics prior. This process unlocks the integration of in-the-wild exocentric data for egocentric world model training. We show that training whole-body action-conditioned egocentric world models with our converted data significantly improves both prediction quality and downstream planning performance, where we infer the sequence of body poses needed to achieve a visual goal state. Our approach paves the way to enlist arbitrary in-the-wild videos for building powerful egocentric world models, furthering applications in robot planning and augmented-reality guidance.
BibTeX
@misc{tran2026egoexo,
  author = {Danny Tran and Roberto Martín-Martín and Kristen Grauman},
  title = {{EgoExo-WM: Unlocking Exo Video for Ego World Models}},
  year = {2026},
  eprint = {2605.15477},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.15477},
}
Figure