VREgo: Consumer VR Headsets for Learning Robot Manipulation from Egocentric Demonstrations

Liuchuan YuarXiv 2026 · arXiv

Learning from human demonstrations (LfD) is a powerful way to teach robots complex manipulation tasks. However, it usually requires expensive research-grade hardware like Project Aria glasses or cumbersome teleoperation setups. VREgo changes the game by showing that consumer VR headsets, like the PICO 4 Ultra, are not just for gaming—they are excellent tools for robot training.

TL;DR

VREgo uses consumer VR headsets to record egocentric human demonstrations for robot manipulation, eliminating the need for specialized research hardware or cloud-based processing. By leveraging on-device stereo vision and introducing task-specific execution rules, the authors improved the success rate of a complex two-stage stacking task from a baseline of 2% to a staggering 93.8% in simulation.

What Problem Does This Paper Address?

Current state-of-the-art egocentric learning methods often rely on monocular (single-camera) research glasses. Because a single camera can't natively see depth, operators must perform a "walk-up"—a specific head movement used to triangulate object positions. Furthermore, these systems often depend on slow, cloud-based processing services to estimate head and hand poses.

Beyond data collection, multi-step tasks (like stacking blocks) are notoriously difficult for imitation learning. Small errors in timing or positioning accumulate, leading to "copycat" behaviors where the robot mimics the human's pauses so well that it never actually releases the object it's holding.

Fig. 1: VREgo trains robot manipulation policies from egocentric demonstrations recorded on consumer VR headsets, here the PICO 4 Ultra. Left: one frame of a two-move demonstration (the red block onto the tray, then the green cube onto the block’s studs), as the headset’s two see-through cameras recorded it; the headset tracked the wearer’s head and hand on the device, and the wearer stood at the table from the first frame. Middle: the same frame as a training image, with the arm removed, a virtual gripper drawn at the tracked hand, and keypoints on the tray, the block and the cube. Right: in simulation, the baseline policy trained on 40 such demonstrations sets the cube on the block in an evaluation layout that no demonstration recorded. The baseline met both placement criteria in 2 of 100 evaluation layouts; after the changes of Section 5.4, 93.8 % of episodes met them.

Methods

VREgo instantiates a pipeline on the PICO 4 Ultra headset. Key technical contributions include:

  • Stereo Localization: Unlike monocular systems, VREgo uses the headset's two see-through cameras to instantly triangulate object keypoints. This allows the operator to start a demonstration while standing still at the table, removing the "walk-up" requirement.
  • Adaptive Preprocessing: The system calculates grasp thresholds for every individual recording based on fingertip distance, making it robust to different operators and tracking noise. It also uses SAM 2 to efficiently propagate segmentation masks across video frames.
  • Trajectory Rewriting: Human hand movements are "bursty" and often approach objects from awkward angles. VREgo rewrites these into "gripper-friendly" paths—adding vertical clearances and centering grasps on objects—before training.
  • Execution Rules: To fix closed-loop failures, the authors added rules like "Commitment" (ensuring a planned release actually finishes) and "Snapped Grasps" (aligning the gripper to the estimated object center).
Comparison of recording protocols: monocular walk-up (Aria) vs. stereo standing start (VREgo).

Datasets

The researchers evaluated VREgo on two tabletop tasks:

  1. Task 1 (Cube on Tray): A single-move task used to compare the PICO 4 Ultra against the research-standard Project Aria glasses across 41 shared layouts.
  2. Task 2 (Two-Stage Stacking): A more complex task (block onto tray, then cube onto block) recorded with the PICO alone.
  3. Evaluation: Both were tested on 100 unseen layouts in the MuJoCo simulator to assess generalization.

Key Insights and Ideas

  • Stereo > Motion: Instant stereo triangulation is significantly more accurate than triangulation from head motion. Static points were located with an error of only 1.1–4.3 mm using stereo, compared to up to 925 mm when relying on head motion alone.
  • The "Replanning Trap": The authors discovered that policies often failed because they predicted a release at the very end of their 50-frame horizon. Since the loop executes only 8 steps before predicting again, and the human stood still during those steps, the next prediction would look identical—effectively deferring the release forever.
  • Inpainting Matters: Removing the human arm from training images and replacing it with a virtual gripper (as seen in Figure 3) is crucial for the policy to focus on the objects rather than the operator's skin.
The VREgo preprocessing pipeline: segmentation, arm removal, and keypoint drawing.

Key Results

  • Device Parity: On the single-move task, PICO-trained policies achieved success rates (60–74%) comparable to those trained on Aria data (73–78%), despite the PICO data being 10x smaller in file size and requiring no cloud processing.
  • Stacking Success: For the difficult two-stage stacking task, the baseline policy was a near-total failure (2% success). By applying trajectory rewriting and execution rules, the success rate soared to 93.8%.
  • Efficiency: Preprocessing a PICO recording took roughly 8.3 minutes on a single GPU, making the workflow highly accessible for rapid iteration.
Success rates on unseen layouts as the number of training demonstrations increases.

Limitations and Future Work

While impressive, all results were obtained in simulation. The simulator provides "perfect" object masks and removes the arm by simply not rendering it, which bypasses the challenges of real-world visual noise and inpainting artifacts.

Additionally, some of the execution rules are task-specific, requiring knowledge of object heights or the specific sequence of moves. Future work will likely focus on making these rules more general and, crucially, testing the trained policies on physical robot hardware to see if the "sim-to-real" gap can be bridged.

Summary updated

Abstract

VREgo trains manipulation policies from demonstrations recorded on consumer VR headsets; we instantiate it with EgoKit on one such headset, the PICO 4 Ultra. On-device head and hand tracking and stereo object localisation allow the operator to begin a demonstration at the table. We adapt HumanEgo's preprocessing to estimate grasp thresholds per recording, update object positions after release, and propagate segmentation masks between keyframes. We evaluate policies in simulation on two tabletop tasks recorded by one operator. For cube placement, we compare PICO and Project Aria recordings of the same 41 layouts. With 40 training demonstrations, the PICO policies placed the cube on the tray in 67, 74, and 60 of 100 evaluation layouts across three training seeds. The corresponding Aria results were 73, 78, and 0. For a two-stage stacking task recorded with the PICO alone, the baseline placed the block on the tray in 94 of 100 layouts and met the end-of-episode criterion for both placements in two. In one diagnostic episode, repeated replanning deferred release beyond the eight executed steps. Demonstration replays also showed that the open gripper could displace the cube before grasping it. After adapting the training trajectories and adding task-specific execution rules that use simulator object positions, the system met both placement criteria in 93.8% of episodes across four evaluation noise seeds. The same 100 layouts informed rule development and evaluation. VREgo is available at https://vrego.chuange.org/.
BibTeX
@misc{yu2026vrego,
  author = {Liuchuan Yu},
  title = {{VREgo: Consumer VR Headsets for Learning Robot Manipulation from Egocentric Demonstrations}},
  year = {2026},
  eprint = {2611.07272},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07272},
}
Figure