TL;DR
Robotic imitation learning policies trained from a single egocentric viewpoint degrade sharply when the robot’s base moves or rotates even slightly. EgoDemoGen solves this by generating kinematically retargeted joint actions alongside photorealistic, temporally coherent novel-view videos synthesized by a diffusion-based repair model (EgoViewTransfer). By training on these generated demonstration pairs, manipulation policies achieve double-digit success rate improvements on both novel and standard egocentric viewpoints in simulation and real-world experiments.
What Problem Does This Paper Address?
End-to-end visuomotor policies (such as ACT and Vision-Language-Action models like $\pi_0$) excel at complex manipulation tasks but remain brittle to viewpoint shifts. In egocentric setups (where cameras are mounted on the robot's base or torso), moving the robot's base causes a mechanical coupling problem: changing the viewpoint fundamentally alters both the camera perspective and the coordinate frame of the robot's manipulators.
Existing novel-view synthesis techniques (like 3D Gaussian splatting or image-to-image diffusion) synthesize new views while retaining the original actions, creating a severe visual–action mismatch. Conversely, motion retargeting pipelines traditionally focus on third-person setups and ignore camera shifts. Robotic policies therefore need demonstration generation pipelines that simultaneously adapt visual observations and robot actions into the new egocentric reference frame.
Egocentric View Shift (Δx, Δy, Δθ)
│
┌──────────────┴──────────────┐
▼ ▼
[Action Retargeting] [Observation Synthesis]
FK -> Base Transform Reproject Scene + Render Robot
-> IK Trajectory │
│ ▼
│ [EgoViewTransfer]
│ Diffusion Video Repair
│ │
└──────────────┬──────────────────────┘
▼
Paired Novel Demonstrations
(Observation, Action)
Methods
EgoDemoGen generates paired demonstrations $(\tilde{V}, \tilde{Q})$ for novel base displacements $(\Delta x, \Delta y, \Delta\theta)$ through two synchronized stages:
- Kinematics-Based Action Retargeting:
- Computes original end-effector poses across time steps via forward kinematics (FK).
- Transforms poses into the novel robot base frame.
- Solves trajectory-level inverse kinematics (IK) using Levenberg-Marquardt optimization with tolerance escalation and restart handling to yield feasible joint trajectories $\tilde{Q}$.
- Novel Egocentric Observation Generation (EgoViewTransfer):
- Scene Video Preparation: Masks out the robot from source RGB-D frames, reprojects the background scene to the target viewpoint, and fills occlusion holes.
- Robot Video Rendering: Uses the retargeted actions $\tilde{Q}$ and the robot's URDF model to render a robot-only execution video from the novel camera perspective.
- Generative Video Repair: Directly merging reprojected scenes and rendered robots produces glaring seam artifacts and temporal flicker. The authors finetune an image-to-video diffusion backbone (CogVideoX-5b-I2V) with 48 input channels to fuse both streams into a seamless, photorealistic video $\tilde{V}$.
- Self-Supervised Double Reprojection: To train EgoViewTransfer without ground-truth novel-view videos, the authors reproject source frames to a random view and then back to the source frame. This synthetically induces occlusion artifacts while preserving source ground-truth labels for diffusion training.
Datasets
The authors evaluate EgoDemoGen across simulated and physical bimanual platforms:
- Simulation (RoboTwin 2.0): Uses dual Piper arms. EgoViewTransfer is finetuned on 390 clean demonstrations across 39 manipulation tasks. Evaluation focuses on three core tasks: Lift Pot, Handover Mic, and Shake Bottle (100 demonstrations per task).
- Real-World (Mobile ALOHA): Uses a dual-arm platform equipped with RealSense D435i cameras. EgoViewTransfer is finetuned on ~1,000 multi-task episodes across 13 diverse tasks (with MoGe-2 estimating missing metric depth). Evaluation covers Pick Up Bowl, Place Bowl to Basket, and Place Bowl to Plate (50 teleoperated demonstrations each).
Key Insights and Ideas
- Action-Observation Co-adaptation is Mandatory: In egocentric manipulation, visual augmentation without corresponding IK retargeting drops replay success from 78.3% down to 8.3%. Both signals must be transformed synchronously.
- Double Reprojection Bypasses Data Collection Bottlenecks: Simulating viewpoint artifacts via forward-and-backward reprojection enables training high-capacity diffusion models completely self-supervised, eliminating the need for multi-camera capture rigs.
- Novel Views Act as In-Distribution Regularization: Incorporating synthesized novel-view demonstrations does not sacrifice performance on the standard view; instead, it improves standard-view success rates by expanding policy robustness against calibration jitter and minor pose variations.
- 1:1 Data Mixing Knee: Performance increases steadily as generated pairs are mixed into training, showing a distinct knee at a 1:1 original-to-synthetic ratio before yielding diminishing returns at 1:3.
Key Results
- Simulation Policies (ACT):
- Novel View: Average success jumps from 14.7% (standard baseline) and 15.3% (naive video merging) to 30.0% with EgoDemoGen (+15.3% absolute gain).
- Standard View: Success climbs from 78.0% to 80.7%.
- Real-World Dual-Arm Policies ($\pi_0$):
- Novel View: EgoDemoGen achieves 62.5% average success, outperforming the 50-demonstration baseline (36.7%, a +25.8% boost) and naive composition (43.3%).
- Standard View: Climbs from 60.0% to 78.3% (+18.3% absolute gain).
- Video Fidelity: EgoViewTransfer slashes Fréchet Video Distance (FVD) in half compared to naive composition (460.62 $\to$ 211.99 in simulation; 896.46 $\to$ 148.61 on real-world data).
Limitations and Future Work
- Narrow Perturbation Bounds: Novel viewpoints are limited to small perturbations ($\Delta x, \Delta y \in [\pm 0.1\text{m}]$, $\Delta\theta \in [\pm 10^\circ]$) to ensure IK solutions remain feasible within manipulator workspaces.
- Inference Chunking Overhead: EgoViewTransfer generates video in 49-frame windows, requiring temporal segment stitching for long-horizon episodes.
- Hardware Movement Constraints: In real-world ALOHA experiments, forward base translation was restricted ($\Delta x \in [-0.1, 0.0\text{m}]$) due to collision risks with the workspace table.
- Depth Map Dependency: High-quality reprojection requires reliable depth sensors or monocular depth estimation, which can introduce artifacts when handling transparent or reflective objects.
Summary updated