TL;DR
BiDemoSyn is a simulator-free framework that synthesizes thousands of contact-rich, physically feasible bimanual robot demonstrations from just a single real-world human demonstration. By decomposing tasks into invariant core motions and variable, vision-aligned adjustments, it generates high-fidelity training data in about 5 seconds per trajectory, enabling visuomotor policies to achieve state-of-the-art dual-arm manipulation performance without sim-to-real transfer gaps.
What Problem Does This Paper Address?
Training visuomotor policies (e.g., Diffusion Policy) for complex, dual-arm robotic manipulation typically demands massive datasets of expert demonstrations. Today's practitioners face an agonizing trade-off:
- Human teleoperation / kinesthetic teaching produces physically grounded, high-fidelity data but is prohibitively slow and labor-intensive (taking ~90 seconds per demo).
- Simulation-based generation scales automatically but suffers from visual and dynamic sim-to-real gaps that cause failures in fine-grained, contact-rich tasks.
- Data augmentation via point-cloud warping creates severe visual distortion artifacts that degrade perceptual learning.
The central question is: How can we scale up real-world bimanual demonstrations without relying on simulators, manual replays, or visual distortions?
Methods
BiDemoSyn synthesizes data directly in the physical domain using a three-stage pipeline based on a single seed demonstration:
[1-Shot Kinesthetic Demo]
│
▼
1. Deconstruction ──► Breaks task into Invariant Blocks (e.g., pouring, twisting)
│ & Variable Blocks (e.g., initial reaching/grasping).
▼
2. Vision Alignment ─► VFM perception (Florence-2 + SAM2 / YOLOE) segments new
│ object instances; PCA on depth finds 6D poses to update grasps.
▼
3. Trajectory Opt. ──► Solves IK reachability, verifies collision avoidance, and
offsets endpoints by bounding box size (Δl, Δw, Δh).
│
▼
[Scaled Real-World Dataset (D_syn)] ──► Train Bimanual Diffusion Policy (DP3 / EquiBot)
- Semantic Deconstruction: Breaks the original trajectory into coarse blocks and minimal units called Atomic Execution Primitives (AEPs). Crucially, it categorizes them into invariant blocks (structural actions like twisting a cap or tilting a cup) and variable blocks (reaching and grasping, which depend strictly on initial object layout).
- Vision-Guided Initial Frame Alignment: When new objects are placed on the table, vision foundation models (Florence-2 and SAM2) detect and segment the items. Using 3D image moments and PCA on depth point clouds, the system calculates the object’s 6D pose ($P_{\text{obj}} \in SE(3)$) and computes a rigid transformation to map the seed grasp pose to the novel scene.
- Hierarchical Trajectory Optimization: Before accepting a synthetic path, inverse kinematics (IK) validates joint reachability and a collision checker ensures motion safety. Grasp offsets are scaled to adjust for differences in instance dimensions ($\Delta l, \Delta w, \Delta h$).
- Operator UI Interface: An augmented-reality UI overlays digital grid targets on the live camera stream, letting a human quickly reposition objects while the system validates and saves feasible synthetic trajectories in roughly 5 seconds.
Datasets
The authors evaluated BiDemoSyn on a dual-arm real-robot testbed (two 6-DoF AUBO-i5 arms with parallel grippers and a Kingfisher stereo camera) across six contact-rich bimanual tasks, along with two extended long-horizon tasks:
plugpen: Aligning and inserting a pen body into a cap ($\pm$sub-millimeter tolerance).inserting: 180° cup flip synchronized with vertical pen insertion.unscrew: Left arm stabilizes bottle while right arm untwists the cap.pouring: Tilting a water bottle into a mug held by the opposite arm.pressing: Dispensing shampoo/water while catching liquid with a tilted cup.reorient: Picking up a flat shovel/spoon, mid-air handover, and 180° flip.- Long-Horizon Additions:
reorient + unscrewandunscrew + pouring.
Across all tasks, the framework synthesized between 1,000 and 7,700 demonstrations per task directly on the real setup.
Key Insights and Ideas
- Functional Invariance vs. Contextual Adaptability: Complex manipulation doesn't require reinventing the entire motion for every scenario. Core physical interactions (e.g., the liquid-pouring arc or cap twisting) stay invariant; only the entry waypoints and spatial endpoints need geometric adaptation.
- Eliminating "Sim-to-Real" by Staying in Reality: Rather than synthesizing pixels in simulation or warping 3D point clouds with artificial distortions, BiDemoSyn collects real RGB-D frames of real objects on the table and adjusts the trajectory analytically.
- Behavioral Noise is Not Generalization: Recent work often adds random trajectory jittering to diversify data. The authors show that random low-level jitter often hurts policy convergence. Systematic spatial diversity (translating and rotating objects across a grid) combined with instance shape diversity is what actually drives out-of-distribution robustness.
Key Results
- Synthesis Speed & Usability: BiDemoSyn generates a demonstration in ~5 seconds, vastly outpacing physical robot replay (YOTO at 42s) and manual teleoperation (91s), while avoiding the perspective distortions of 3D point-cloud editing (DemoGen).
- Manipulation Success Rates: When training an equivariant diffusion policy (EquiBot):
- In-Distribution (ID): BiDemoSyn achieved an average success rate of 86.7% across all six tasks, outperforming DemoGen (68.9%), YOTO (71.1%), and zero-shot baselines like ReKep (42.2%) and ODIL (54.4%).
- Out-of-Distribution (OOD): On novel object instances and unseen positions, BiDemoSyn maintained 66.7% success, far ahead of DemoGen (40.6%) and YOTO (47.8%).
- Data Scaling Laws: Performance scaled steadily with synthetic data volume (e.g., DP3 success increased from ~25% at 550 demos to >80% at 17,000 aggregate demos), proving that synthetic trajectories obey empirical imitation scaling laws.
Limitations and Future Work
- Quasi-Static Assumption: Trajectory modulation assumes quasi-static conditions and cannot yet handle dynamic, ballistic actions (e.g., catching or dynamic throwing).
- Deformable Objects & Complex Contacts: Tasks involving multi-stage interdependent friction and deformation (like tying shoelaces or folding flexible cloth) are currently out of scope.
- Symmetric / Textureless Ambiguity: 47% of pipeline failures stem from orientation estimation errors (e.g., confusing concave vs. convex sides of identical reflective spoons) or pre-grasp collisions on irregular geometries.
- Hardware Limitations: Binary grippers lacking tactile/torque feedback restrict fine-grained impedance control, which the authors aim to address in future work through adaptive contact modeling and dynamic visual servoing.
Summary updated