VBot Exo2Ego Policy: Lightweight Data Collection and Exocentric-to-Egocentric Video Generation

Tong Rao, Zhihao Zhang, Yuntian Wang, Siyu Zhou, Hailong Qin, Yinan YuarXiv 2026 · arXiv

TL;DR

VBot Exo2Ego Policy is a generative framework that transforms third-person "exocentric" (EXO) videos into first-person "egocentric" (EGO) views. By leveraging abundant third-person footage—from crowdsourced recordings to Internet videos and classic movies—it provides a scalable way to generate high-fidelity training data for robot manipulation and human-activity analysis.

What Problem Does This Paper Address?

Egocentric video is the "gold standard" for learning manual operations because it captures the exact visual context of an actor’s hands and the objects they manipulate. However, collecting diverse EGO data is a bottleneck: it requires specialized head-mounted hardware and significant participant effort.

In contrast, exocentric video is everywhere. The challenge lies in the "viewpoint gap." Converting a third-person view into a first-person one requires the model to hallucinate missing information (like the actor's exact hand pose or occluded object surfaces) while maintaining temporal consistency and action fidelity. Existing methods often struggle with complex multi-person scenes, camera cuts, and the fine-grained details of hand-object interactions (HOI).

Methods

The authors introduce a comprehensive system consisting of a data collection pipeline and a novel video generation model.

1. Lightweight Data Collection: They developed a mobile app for LiDAR-equipped iPhones that allows for synchronized EGO/EXO capture. This provides the "ground truth" pairs needed to train the model on cross-view correspondences without requiring expensive professional motion-capture rigs.

2. VBot Exo2Ego Policy Architecture: Built on a pretrained video diffusion transformer (MiniMax-H3), the model incorporates several clever conditioning mechanisms:

  • Stochastic Camera Conditioning: During training, the model is fed target camera geometry via Plücker rays. Crucially, this conditioning is "annealed" (gradually removed) during training. This allows the model to learn spatial mappings while remaining functional at inference time on uncalibrated videos where camera parameters are unknown.
  • Source Reasoning: The system uses a vision-language model (Qwen3.8-VL) to "reason" about the EXO clip, generating a text description of the actions and identifying the intended actor.
  • Auxiliary Supervision: To ensure the generated hands and objects look realistic and are anatomically correct, the model uses a joint training objective. It predicts semantic masks and hand keypoints (using ACE features) alongside the RGB video.
Figure 2. VBot Exo2Ego Policy architecture. Solid paths generate EGO RGB; dashed paths introduce geometric, semantic, and hand supervision during training.

Datasets

The study utilizes a mix of internal and public data:

  • VBot Echo Capture: A new dataset of synchronized EGO/EXO pairs collected via their iPhone pipeline (summarized in Figure 3).
  • Hand Supervision Benchmarks: They leverage established datasets like HOT3D, ARCTIC, and Re:InterHand to refine the hand-structure supervision.
  • In-the-Wild Evaluation: The model was tested on diverse Internet footage, including clips from Raiders of the Lost Ark, Star Wars, and Charlie Chaplin films, to demonstrate generalization.
Table 1. Data sources and uses. Recorded and generated views have distinct provenance.

Key Insights and Ideas

  • Geometry as a Training Scaffold: The most striking idea is using calibrated camera rays as a "scaffold" that is eventually kicked away. By injecting geometry stochastically, the model learns the concept of a viewpoint shift without becoming dependent on precise metadata at test time.
  • Semantics Drive Fidelity: The authors found that simply asking a model to "make it look like a first-person view" isn't enough for robotics. By forcing the model to also predict semantic masks, the internal features become much better at respecting object boundaries and hand-object contact points.
  • Handling the "Director's Cut": Real-world videos have camera cuts. The model uses "multi-shot adaptation"—training on clips that switch between different EXO views of the same action—to learn how to maintain a consistent EGO perspective even when the source video jumps.

Key Results

VBot Exo2Ego Policy significantly outperforms existing baselines like EgoX:

  • Visual Quality: It achieved a 3.23 dB gain in PSNR and a 37.6% reduction in FVD (Fréchet Video Distance) compared to the released EgoX model, indicating much higher image fidelity and temporal smoothness.
  • Interaction Accuracy: Hand structure recovery improved dramatically. The PCK@5% (Probability of Correct Keypoint within a 5% threshold) jumped from 11.52% to 20.02% when using the proposed hand supervision strategies.
  • Generalization: As shown in Figure 1, the model successfully generates plausible EGO views from highly varied sources, including low-light environments and complex cinematic shots.
Figure 1. From observed operations to first-person video. Selected EXO inputs and generated EGO frames.

Limitations and Future Work

Despite the progress, the authors note several hurdles:

  • Ambiguity: If an EXO view completely hides a hand, the model must "guess" the pose. While the results are plausible, they may not always match the unobserved reality.
  • Complex Editing: While the model handles simple camera cuts, highly stylized editing with non-linear time skips remains a challenge.
  • Future Directions: The team suggests expanding the model to handle even broader varieties of edited video and improving the uncertainty estimation for the "hallucinated" regions of the EGO view.

Summary updated

Abstract

We present VBot Exo2Ego Policy, a video generation model that turns third-person videos into first-person observations of a selected actor, offering a practical way to expand first-person operation data from abundant real-world footage. To support this goal, we introduce a lightweight data collection pipeline and mobile app that support first-person recording, third-person recording, and synchronized capture of both views. This setup combines accessible single-view collection with paired RGB and camera geometry for learning cross-view correspondence. Building on a pretrained video diffusion transformer, our model incorporates geometric guidance and auxiliary supervision from semantic masks and hand keypoints to facilitate optimization and improve the fidelity of hand-object interactions. Camera geometry is injected stochastically during training and gradually removed, allowing inference without target-camera calibration. To accommodate the diversity of Internet videos, we combine source-video perception and reasoning, optional actor references, and multi-shot adaptation to identify the intended actor and follow an operation across camera cuts. The collection system and generation model provide a foundation for expanding first-person operation data across tasks, scenes, and participants.
BibTeX
@misc{rao2026vbot,
  author = {Tong Rao and Zhihao Zhang and Yuntian Wang and Siyu Zhou and Hailong Qin and Yinan Yu},
  title = {{VBot Exo2Ego Policy: Lightweight Data Collection and Exocentric-to-Egocentric Video Generation}},
  year = {2026},
  eprint = {2611.16417},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16417},
}
Figure