When a person wears a camera while cooking, their head never holds still. It bobs, turns, and translates as they reach for things. Nearly every method that teaches robots from such footage treats this camera motion as a nuisance. Models either absorb it through sheer scale or assume the demonstration view already matches the robot's camera. This proposal treats the head motion as a free signal. When a camera translates, nearby points shift across the image more than distant ones. This effect, called motion parallax, is what view-dependent displays exploit to make flat screens look three-dimensional. It is also enough to recover the 3D layout of the scene and the camera's own path. We use it to lift the hand and object motion in each human clip into a stable 3D world frame. We then re-project that motion into the fixed camera pose of the target robot, so that every human demonstration is re-expressed as if the robot's own camera had watched it.
The payoff is practical. Current human-to-robot systems fail most often at contact, and they fail because they get depth wrong. In Zero-Shot Robot Manipulation from Passive Human Videos, which predicts palm trajectories from egocentric video and maps them to a Franka arm, roughly 40% of real-world failures were contact misses caused by monocular depth errors. A policy trained on motion that has been geometrically moved into the robot's viewpoint, with depth taken from the demonstrator's own parallax, should reach and grasp more accurately than one trained on raw egocentric footage. Put in two sentences: we use the demonstrator's head motion to reconstruct each human demonstration in 3D, then re-render its point trajectories from the robot's camera. We train a trajectory-predicting policy on these re-rendered examples, which should reduce depth-driven contact failures in human-to-robot transfer.
Core Claim
The claim is that perspective warping is most useful for cross-domain motion transfer as a training-time data operator applied to geometry, not as a visual effect applied to pixels. Concretely, it means re-projecting parallax-lifted 3D point trajectories into the robot's camera frame. Two consequences follow. First, a track-based policy pretrained on canonicalized human trajectories should outperform the same policy pretrained on raw egocentric trajectories when human and robot cameras differ. Second, the gain should come mostly from fewer contact misses, which we attribute trial by trial.
The core re-projection is simple. Let $x_t$ be a tracked pixel in egocentric frame $t$ and $d_t$ its parallax-derived depth. Let $T_{w \leftarrow c_t}$ be the estimated pose of the head camera in a world frame, $T_{r \leftarrow w}$ the robot camera's pose relative to that world, and $\pi_{K_r}$ the projection through the robot camera's intrinsics $K_r$. The canonicalized track point is
$$x^{r}_t = \pi_{K_r}\big(T_{r \leftarrow w}\, T_{w \leftarrow c_t}\, d_t K_c^{-1} \tilde{x}_t\big),$$
where $K_c$ is the head camera's intrinsics and $\tilde{x}_t$ is the pixel in homogeneous coordinates. Structure-from-motion recovers depth only up to an unknown global scale. We fix that scale with a metric prior on human hand size. We also report how sensitive results are to this choice.
Relation to Prior Work
Four groups of prior work shape the proposal, each in a different role.
The first supplies the policy we build on. Any-point Trajectory Modeling for Policy Learning (ATM) predicts future 2D trajectories of scene points and feeds them to a lightweight controller, running about 500 times faster than video diffusion. It reports that its performance is sensitive to the static camera pose. We keep ATM's architecture and change only what its tracks are expressed in. That makes ATM the natural base and its camera sensitivity our target.
The second group motivates the gap. GR-1 pretrains a GPT-style transformer on 800,000 Ego4D clips, and SuSIE fine-tunes an image-editing diffusion model on robot and human data to generate subgoal images. Both ingest egocentric footage without correcting its viewpoint. Mitty turns human videos into robot videos with a video diffusion transformer but assumes the two views are already aligned. SWIM closes the viewpoint and embodiment gap empirically, with 30 minutes of robot play. None of these re-projects the demonstration geometrically, and that is the step we add.
The third group supplies warping techniques but not transfer. EPiC redirects the camera of an existing video using optical-flow anchors and visibility masks. 4DStreamCtrl unifies camera and object motion into 3D point tracks for streaming generation. Neither produces robot actions, and 4DStreamCtrl's coarse 32×32 track grid is too sparse to resolve gripper contact. We borrow EPiC's visible-pixels-only anchor idea for one step only: building the robot-view conditioning image the policy sees. Everything that drives action stays geometric.
The fourth group provides the perceptual premise. The IEEE J-STSP 2015 guest editorial summarizes evidence that viewpoint updates driven by head motion are a stronger depth cue than static stereo. We carry that observation from human viewing to machine depth estimation. Radiance Fields for Robotic Teleoperation renders arbitrary viewpoints with Gaussian splatting, but it produces tearing artifacts when the scene moves. That result constrains us: we should not try to reconstruct dynamic manipulation densely. Instead, we reconstruct the static background densely enough to estimate the camera's motion, and treat the moving hand and object only as sparse tracks.
Research Program
The pipeline has three stages.
The lifting stage runs on each egocentric clip. It first masks the hands and manipulated objects using hand-object contact detection of the kind 100 Days of Hands annotates. It then tracks dense points and estimates the camera path and depth from the static background alone, since background parallax is uncontaminated by the manipulation. Finally, it assigns depth to hand and object tracks by triangulating over short windows, anchored at contact to nearby static surfaces.
The canonicalization stage samples a robot camera pose from the distribution of real mounts on the target robot. It places that camera relative to the reconstructed tabletop plane and re-projects all tracks into it using the equation above. It also builds a robot-view conditioning image by warping the visible background pixels and filling the regions that become newly visible. We test two fill methods: a simple inpainter and an EPiC-style anchor-conditioned generator.
The learning stage pretrains an ATM track transformer on the canonicalized human tracks. It then trains the track-to-action controller on a small set of action-labeled robot demonstrations, exactly as ATM does.
Three analyses test whether the mechanism is what we claim. The first measures depth accuracy of the parallax lifting against RGB-D ground truth. The second measures track error in the robot frame. The third labels every robot failure by cause: contact miss, wrong target, or other.
Baselines
The do-nothing reference is ATM pretrained on the same human clips with tracks left in the raw egocentric frame, plus ATM's standard behavioral cloning comparison. On LIBERO-Long, ATM reports a 39% success rate against 16% for behavioral cloning. On a real UR5 with human videos, it reports 60–63% against 0–30% for behavioral cloning (ATM). Every variant shares the same human clips, the same robot demonstrations, the same architecture, and the same number of pretraining steps and hyperparameter trials. Only the track coordinates and conditioning image differ.
The strongest baseline for depth and contact is direct 3D palm-trajectory transfer, as in Zero-Shot Robot Manipulation from Passive Human Videos. It reports a 50% real-world success rate with about 40% of failures from depth errors. We reimplement it on our tasks and robot, fed the same clips. Beating it means higher success and a substantially lower share of contact-miss failures, not higher success alone.
The strongest learned-from-video competitor is GR-1-style generative pretraining (GR-1). It reports 85.4% on CALVIN ABC→D, but that setting has no human-to-robot viewpoint shift. We pretrain it on the same raw clips at matched compute.
Two ablations isolate the key ingredient. The first keeps canonicalization but replaces parallax-derived depth with per-frame monocular metric depth. The second keeps parallax depth but skips re-projection, applying only random homography augmentation.
Why not just…?
Why not just train on raw egocentric video with heavy viewpoint augmentation? GR-1 shows raw Ego4D pretraining already helps downstream policies. Random crops and homographies are cheap. However, homographies cannot reproduce parallax, meaning the depth-dependent shift of near and far points, so they cannot simulate a camera that has actually moved. The homography ablation above tests exactly this. If it matches full canonicalization, our claim is wrong, and we will say so.
Why not just use a strong monocular metric depth network instead of parallax? This is the most serious threat, because depth networks keep improving and need no camera motion. Our bet is that multi-frame triangulation is more consistent over time, and that contact errors are errors in relative depth between hand and object, which a single frame estimates poorly. The monocular-depth ablation, combined with depth accuracy against RGB-D and the contact-miss attribution, settles the question directly. It can come out either way, and a tie would mean the geometric re-projection, not parallax, carries the gain.
Why not just translate the whole human video into robot video, as Mitty does, or redirect its camera, as EPiC does? Mitty reports 84.5% task success on its human-to-robot benchmark, but that is measured on generated video. It still needs a separate controller, and diffusion generation runs at seconds to minutes per plan. Redirecting the full manipulation video with EPiC would require hallucinating the moving hand, which is precisely where Radiance Fields for Robotic Teleoperation finds dynamic content breaks reconstruction. We include an EPiC-redirected-video variant, with tracks extracted from the redirected output, at matched data. This tests whether pixel-level warping adds anything beyond the geometric version.
Why not just calibrate with robot play, as SWIM does? SWIM reaches 80% on six real tasks after 30 minutes of play. Play is an orthogonal way to close the gap. We give our method and a SWIM-style play-calibrated ATM the same 30-minute budget, which shows whether canonicalization removes the need for play or adds to it.
Proposed Experiments
The first experiment is a controlled simulation testbed. It is needed because no public benchmark pairs moving-head human video with fixed-camera robot evaluation; no reported result was found on Lacuna for LIBERO with moving-camera actionless video. We re-render LIBERO demonstrations as actionless videos from a simulated head-mounted camera that follows recorded human head trajectories. Robot action labels come only from the fixed robot camera, using ATM's small labeled-demo regime. Because simulation gives exact depth and camera poses, we can separately measure error from lifting, from scale, and from canonicalization. The decisive check is whether canonicalized ATM closes most of the gap between raw egocentric ATM and an oracle ATM pretrained on videos rendered from the robot camera itself.
The second experiment moves to real hardware. We pretrain on Epic-Kitchens plus about an hour of head-mounted recordings in our lab kitchen, then deploy on a Franka arm with a fixed camera on six pick-and-place and articulated-object tasks matched in type to Zero-Shot Robot Manipulation from Passive Human Videos. Metrics are success rate, robot-frame track error against RGB-D ground truth, and the share of failures from contact misses. The decisive check is causal. If the success gain over raw-egocentric ATM comes predominantly from fewer contact misses, and the monocular-depth ablation recovers less of it, then the demonstrator's parallax is doing the work we claim.
A final stress test varies the offset between the human's head height and the robot camera's mount. It should show that the benefit grows with viewpoint mismatch; a flat curve would falsify the mechanism.
References
- Zero-Shot Robot Manipulation from Passive Human Videos
- Any-point Trajectory Modeling for Policy Learning
- Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation
- Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models
- Mitty: Diffusion-based Human-To-Robot Video Generation
- Structured World Models from Human Videos
- EPiC: Efficient Video Camera Control Learning with Precise Anchor-Video Guidance
- 4DStreamCtrl: Interactive Video Generation with Online 4D Control
- Introduction to the Issue on Interactive Media Processing for Immersive Communication
- Radiance Fields for Robotic Teleoperation