Robots learn to manipulate from demonstrations, and demonstrations are expensive. Human videos are cheap, and the most useful ones are egocentric (filmed from the wearer's head). Methods that pretrain robot vision on such footage ask the network to predict where the human hand will touch the object next. They express that answer in the camera's image, so the supervision depends on where the camera happened to be. We propose to express the same answer in a coordinate frame attached to the object itself, measured in meters: "the thumb lands 4 cm from the centroid along the handle axis." Such a target means the same thing from any viewpoint, and it has real physical scale.
This one change touches three problems. The supervision stops caring about camera placement, so third-person instructional video becomes as usable as scarce egocentric video. The encoder is pushed to infer object geometry independently of the camera, the frame-anchoring that perspective-taking benchmarks identify as the core spatial failure of vision-language models. A robot whose cameras sit in different places from the human's head should then inherit the prior with less loss. The claim is testable: if object-frame targets do not make downstream policies more robust to camera-viewpoint shifts than camera-frame targets built from identical data, the idea is wrong.
Core Claim
Pretraining a visual encoder to predict future hand-object contact points and approach paths in a metric, object-centered frame yields robot manipulation features that hold up better than image-plane or camera-frame contact prediction when the robot's camera viewpoint differs from training. Because the target ignores the camera, the same objective can absorb exocentric video without the 2D cropping tricks the egocentric-transfer literature relies on.
Where This Sits in Prior Work
The method closest to ours is MAPLE. It mines about 1,590 hours of Ego4D for pre-contact frames and trains a ViT (vision transformer) to forecast 3D hand pose and contact points, then feeds the frozen or fine-tuned encoder to a diffusion policy. Its contact targets are normalized image-plane coordinates with relative hand pose. It therefore supplies our data pipeline and our downstream protocol, and our single change is the coordinate frame of the target.
Learning Precise Affordances from Egocentric Videos (GAT / Aff-Grasp) also mines contact from egocentric video. It projects contact back onto a pre-contact frame, segments the affordance with SAM, and shows that DINOv2 features need injected pseudo-depth to stop attending to color. That finding motivates us: geometry has to be forced into the representation. GAT stays in the image plane and drops the approach trajectory. We keep both the 3D path and metric scale.
Cortical Policy motivates the viewpoint question from the robot side. Its static-camera stream uses VGGT 3D features with a cross-view consistency loss, and its wrist stream is pretrained on 3,600 synthetic egocentric videos rather than human footage. It reaches 81.0% on RLBench but 24% zero-shot on unseen task geometry. This suggests that consistent geometry alone, without human contact priors, does not generalize.
On the egocentric-scarcity side, Unlocking Exocentric Video-Language Data crops third-person video around the hands, while Exo2EgoDVC and Viewpoint Rosetta Stone remove viewpoint information adversarially or translate features across views. These methods constrain our design. They buy viewpoint invariance by discarding the metric, view-dependent geometry that a gripper needs. Cropping also leaves a demonstrator facing the camera upside-down relative to an egocentric actor. Our target is invariant because it lives on the object, and it keeps metric geometry.
Grounded Affordance from Exocentric View transfers affordance heatmaps across views but never reaches 3D or control. SemFM transfers affordances between object instances on unscaled meshes and needed manual depth alignment to set gripper width on the real robot. That is a direct warning that scale must be in the target, not added at deployment.
The spatial-reasoning literature supplies the diagnosis. Vision Language Models See What You Want but not What You See finds no model above chance (about 29%) on Level-2 perspective-taking across 37 models. Seeing Through Their Eyes reports 32–38% drops when the reference frame leaves the camera. ViewSpatial-Bench shows that explicit multi-perspective supervision lets a 3B model reach 82.09% against GPT-4o's 34.98%. We take the lesson that frame-anchoring is fixed by supervision expressed in non-camera frames, not by scale. We apply it to manipulation features rather than question answering.
Research Program
Building the targets. For each interaction clip, we take the contact frame and a pre-contact prediction frame using MAPLE's pipeline. We then lift the clip to 3D with an off-the-shelf monocular metric depth estimator and hand mesh recovery. We use camera intrinsics where Ego4D metadata provides them and estimated intrinsics otherwise, and we flag these clips for a noise ablation. The object is segmented with SAM prompted by the contact points, following GAT, and its pixels are back-projected into a partial metric point cloud.
The object frame $O$ has three parts:
- Origin: the centroid of the object point cloud.
- Vertical axis: the gravity direction, from the head-mounted IMU when present, otherwise from an estimated floor or table plane.
- Horizontal axis: the dominant principal axis of the point cloud projected onto the horizontal plane. Its sign is fixed so that the hand's approach has a positive component.
The target is the sequence of fingertip and contact positions over the next second, expressed in $O$. For a transform $T_{O\leftarrow C}$ from camera to object frame, observed fingertip positions $p_t^C$ at future step $t$, and encoder prediction $\hat p_t$ from the prediction frame alone, the pretraining loss is
$$\mathcal{L} = \sum_{t=1}^{H} \left\| T_{O\leftarrow C}\, p_t^C - \hat p_t \right\|_1 .$$
Here $H$ is the number of future steps. The encoder is never shown $T_{O\leftarrow C}$, so it must infer the object's frame from appearance. That inference is the skill we want transferred.
Adding exocentric video. Because the target does not reference the camera, the identical pipeline runs on third-person instructional video. We use the same source pool as Unlocking Exocentric Video-Language Data, but replace cropping with lifting. We measure value per hour against egocentric hours.
Downstream use. The pretrained ViT is frozen, or lightly fine-tuned, inside the same diffusion policy MAPLE uses, with identical robot demonstrations. On RLBench we plug it into the static and wrist streams of a Cortical-Policy-style architecture.
Baselines
The naive reference is a diffusion policy on an ImageNet-initialized ViT trained only on robot demonstrations, alongside frozen DINOv2 and R3M. MAPLE reports 27.9% (DINOv2) and 24.0% (R3M) mean success on its eight simulated dexterous tasks.
The strongest existing human-video prior is MAPLE itself. It reports 31.9% mean success over those eight tasks and 75.1% on Pen reorientation. We reproduce it with the same 1,590 Ego4D hours and 82,100 samples, the same ViT size, the same pretraining compute, and the same downstream demonstrations and tuning budget. Beating it means a higher mean success on the eight tasks, with non-overlapping seed intervals, and especially a smaller drop under camera shift.
The strongest geometry-first competitor is Cortical Policy on RLBench. It reports 81.0% success over 18 tasks, 3.5 points above RVT-2. We run it with its VGGT stream, retrained on our camera-shift splits, under the same demonstration count.
The key ablation is a camera-frame variant. It is trained on the identical lifted 3D trajectories, but the targets stay in metric camera coordinates. A second variant uses object-frame targets with metric scale removed, by normalizing by object extent. Together these separate "3D" from "object-centered" from "metric." If the camera-frame variant matches ours, the proposal's central mechanism has no effect.
Why not just…?
Why not just add depth to MAPLE's targets in camera coordinates? This is the most serious alternative, so it is our main ablation rather than a strawman. Camera-frame 3D targets carry metric information but tie the answer to head pose. Our hypothesis predicts that the two variants tie at the training viewpoint and diverge under viewpoint shift. If they do not diverge, we will report that the object frame adds nothing beyond lifting.
Why not just use viewpoint-consistent 3D foundation features such as VGGT, or DINOv2? Cortical Policy already does this for its static stream and still falls to 24% on unseen task geometry. MAPLE shows DINOv2 trailing a contact-trained encoder. Consistent geometry does not tell the network where hands go. VGGT may still capture most of the viewpoint robustness, so we also test our objective trained on top of VGGT features, to see whether the gains add.
Why not just ask a vision-language model to plan spatial steps, as Spatial Policy does? Spatial Policy uses GPT-4o to write metric displacement tables and reaches 86.7% on Meta-World but 30% on a real SO-100 arm. The perspective-taking results above show these models cannot reliably leave the camera frame. A planner built on that skill inherits the failure we aim to train away.
Why not just generate more simulated robot data, as RoboCasa does? RoboCasa supplies over 100,000 synthetic trajectories, but the file reports no success numbers for it. The sim-to-real collapses reported by Spatial Policy (86.7% to 30%) and Cortical Policy (24% zero-shot) indicate that simulated volume does not supply human contact priors. Simulation and our prior are complementary, and we do not claim to replace it.
Proposed Experiments
The decisive check is a viewpoint-shift protocol. On MAPLE's eight simulated dexterous tasks and on RLBench's 18 tasks, we train policies with the default cameras. We then evaluate with cameras moved by fixed increments of azimuth and elevation, which is a protocol we define, since COLOSSEUM in the file perturbs lighting, texture, and color rather than camera pose. The primary metric is the slope of success rate against camera displacement, compared across:
- our encoder,
- the camera-frame ablation,
- MAPLE,
- DINOv2,
- Cortical Policy.
The claim holds only if our slope is flatter than the camera-frame ablation's at matched in-distribution success.
Exocentric value. We pretrain on equal hours of egocentric-only, exocentric-only, and mixed footage. Under object-frame targets, exocentric hours should approach the downstream value of egocentric hours. Under the camera-frame ablation they should not. A positive result would be direct evidence that the target removes the egocentric data bottleneck rather than routing around it.
Spatial probe. We test whether the representation itself became less camera-anchored. A linear probe on the frozen features predicts object-frame quantities, namely approach direction and contact offset in centimeters. The probe is evaluated across the multiple simultaneous egocentric views of MultiEgo, measuring cross-view prediction disagreement. Lower disagreement than baselines would tie the robotic gains to the frame-invariance diagnosed in the perspective-taking literature. That literature reports no result of this kind for manipulation encoders; no reported result was found on Lacuna for cross-view contact consistency.
Real-robot test. We repeat MAPLE's Wash-the-Dish task on an ORCA hand (MAPLE reports 69%, baselines 0%) with the wrist camera remounted at a new angle. Physical gripper width is set from predicted metric contact offsets with no manual alignment, the step SemFM required by hand.
Lifting-noise control. Every result is reported separately for clips with real and with estimated intrinsics. This separates the value of the object frame from the quality of monocular metric lifting.
References
- MAPLE: Encoding Dexterous Robotic Manipulation Priors Learned From Egocentric Videos
- Ego4D: Around the World in 3, 000 Hours of Egocentric Video
- Learning Precise Affordances from Egocentric Videos for Robotic Manipulation
- Cortical Policy: A Dual-Stream View Transformer for Robotic Manipulation
- Unlocking Exocentric Video-Language Data for Egocentric Video Representation Learning
- Exo2EgoDVC: Dense Video Captioning of Egocentric Procedural Activities Using Web Instructional Videos
- Viewpoint Rosetta Stone: Unlocking Unpaired Ego-Exo Videos for View-invariant Representation Learning
- Grounded Affordance from Exocentric View
- Affordance Transfer Across Object Instances via Semantically Anchored Functional Map
- Vision Language Models See What You Want but not What You See
- Seeing Through Their Eyes: Evaluating Visual Perspective Taking in Vision Language Models
- ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models
- Spatial Policy: Guiding Visuomotor Robotic Manipulation with Spatial-Aware Modeling and Reasoning
- RoboCasa: Large-Scale Simulation of Household Tasks for Generalist Robots
- MultiEgo: A Multi-View Egocentric Video Dataset for 4D Scene Reconstruction