Ego2Sim2Humanoid: Agentic Real-to-Sim for Learning Loco-Manipulation VLAs without Robot Demonstrations

Chen Liu, Nikolaus CorrellarXiv 2026 · arXiv

TL;DR

E2S2H (Ego2Sim2Humanoid) is a novel pipeline that converts simple egocentric recordings from smart glasses into high-fidelity, physically verified training data for humanoid robots. By using an "agentic" real-to-sim process to reconstruct 3D scenes and optimize robot motion in simulation, the researchers enabled a Unitree G1 humanoid to perform complex "loco-manipulation" tasks—like walking to a table and putting on a hat—without a single real-world robot demonstration.

What Problem Does This Paper Address?

Training generalist humanoid robots is currently bottlenecked by data. The gold standard, imitation learning, typically requires teleoperation, where a human operator controls a physical robot in real-time. This is agonizingly slow, expensive, and doesn't scale.

While we have massive amounts of human video data (like egocentric recordings from smart glasses), using it for robots is difficult due to the embodiment gap. A robot has a different height, arm reach, and hand morphology than a human. Simply "copy-pasting" human joint movements onto a robot usually results in the robot missing the object, colliding with the environment, or losing its balance. Furthermore, video lacks the "ground truth" physics labels—like contact forces and joint torques—needed to train robust control policies.

Methods

The authors introduce E2S2H, a multi-stage pipeline that bridges the gap between human observation and robot action:

  1. Agentic Real-to-Sim: Instead of a static script, a "coding agent" processes the RGB-D and head-tracking data from Aria glasses. It reconstructs the 3D scene, fits planes to floors and tables, and builds simplified collision models for objects. Crucially, it "completes" unseen geometry (e.g., the back of a bottle) and identifies contact bands—regions on an object where a grasp is likely to succeed.
  2. Motion Adaptation: The system maps the human's head and hand paths to the robot. It uses bilevel stance optimization to find the best place for the robot to stand so it can reach the object while remaining balanced and collision-free. A grasp planner then translates the "contact bands" into specific finger joint commands for the robot’s unique hand.
  3. Dynamic Correction via RL: Kinematics alone isn't enough for stable walking and grasping. The pipeline trains a reinforcement learning (RL) tracking policy in the reconstructed simulation. This ensures the motion is dynamically feasible and that the robot can maintain a grasp even when the object moves out of its camera's view.
  4. Robot-View Rendering: Finally, the system renders the scene from the robot's onboard camera perspective. This creates a dataset of synchronized images, proprioception, and verified action labels that are perfectly aligned for training a Vision-Language-Action (VLA) model.

Datasets

The researchers collected a new dataset of human demonstrations using Project Aria Gen 2 glasses:

  • Bottle Lift: 219 episodes (walking to a table, grasping a bottle, and lifting it).
  • Hat Wearing: 154 episodes (walking to a table, picking up a hat, and placing it on the head).
  • Filtering: The pipeline acts as a quality filter. Only 189 bottle-lift and 59 hat-wearing episodes passed the rigorous physics and motion checks to be included in the final training set.

Key Insights and Ideas

  • The Embodiment Gap is a Geometric Constraint: The paper argues that you cannot simply retarget motion; you must re-optimize the entire body posture (stance) to ensure the robot's specific limbs can reach the target while staying clear of obstacles.
  • Physics as a Data Filter: By running every converted trajectory through a physics simulator (MuJoCo), the system automatically discards "bad" data where the robot would have dropped the object or fallen over. This ensures the VLA model only learns from successful, physically grounded examples.
  • Semantic Rewards for Blind Phases: For tasks like putting on a hat, the object often leaves the camera's field of view. The "agentic" part of the pipeline designs semantic rewards (e.g., "is the hat coordinate near the head coordinate?") that allow the RL policy to complete the task even without visual feedback.

Key Results

The researchers fine-tuned a GR00T N1.7 (3B) model exclusively on the synthetic data generated by E2S2H. They then tested it on a real Unitree G1 humanoid:

  • Bottle Lift: Achieved a 45% success rate (9/20 trials) in the real world.
  • Hat Wearing: Achieved a 25% success rate (5/20 trials).
  • Comparison: E2S2H significantly outperformed existing baselines like EgoHumanoid, which failed to produce any feasible grasps for the hat-wearing task due to its lack of precise contact modeling and scene reconstruction.

Limitations and Future Work

  • Dynamics Estimation: Parameters like mass, inertia, and friction are estimated from visual priors. For very heavy or oddly balanced objects, the "sim-to-real" gap in dynamics might still cause failures.
  • Rigid Body Assumption: The current pipeline represents objects as rigid bodies or hinged links. It cannot yet handle highly deformable tasks like folding clothes or cutting bread.
  • Success Rates: While 45% is impressive for zero-shot transfer without real-robot data, it is still lower than what can be achieved with high-quality (but expensive) teleoperated demonstrations. Future work will likely focus on improving the fidelity of the "agentic" reconstruction to close this gap further.

Summary updated

Abstract

Imitation learning for visuomotor policies typically relies on teleoperated robot demonstrations, which are costly because every episode requires a trained operator controlling a physical robot in real time through teleoperation interfaces and wearable body-tracking hardware. Egocentric glasses capture human demonstrations without a robot or handheld grippers, yet they track only the head and in-view human hands, and directly copying human motion to a robot with different stature, arm reach and hand morphology fails to produce feasible contacts. \method{} converts egocentric glasses recordings into physically verified robot training trajectories. An agentic real-to-sim stage reconstructs collision and textured visual assets from the recorded RGB-D and head streams, completes unseen surfaces when needed, and predicts target contact bands on the manipulated objects. During motion adaptation, a grasp planner turns each contact band into a feasible robot wrist pose and finger closure, bilevel stance optimization plans collision-free standing postures that keep hand targets reachable, and a whole-body motion generator synthesizes lower-body locomotion and out-of-view arm motion. Two-stage motion correction then blends visibility transitions kinematically and trains a reinforcement-learning tracking policy in physics simulation under agent-designed semantic rewards, completing manipulation phases even when the object leaves the camera view. Rendering the reconstructed scene through the robot's onboard camera along the optimized rollout yields synchronized robot-view images, proprioception and verified action labels. Fine-tuned exclusively on this converted dataset, without real-robot demonstrations, GR00T N1.7 performed loco-manipulation tasks autonomously on a Unitree G1. Project page: https://robonana.github.io/ego2sim2humanoid/
BibTeX
@misc{liu2026ego,
  author = {Chen Liu and Nikolaus Correll},
  title = {{Ego2Sim2Humanoid: Agentic Real-to-Sim for Learning Loco-Manipulation VLAs without Robot Demonstrations}},
  year = {2026},
  eprint = {2611.19196},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19196},
}
Figure