One-Shot Real-World Demonstration Synthesis for Scalable Bimanual Manipulation

Huayi Zhou, Kui JiaSubmission to ICLR 2026 · Rejected

TL;DR

BiDemoSyn is a simulator-free framework that synthesizes thousands of contact-rich, physically feasible bimanual robot demonstrations from just a single real-world human demonstration. By decomposing tasks into invariant core motions and variable, vision-aligned adjustments, it generates high-fidelity training data in about 5 seconds per trajectory, enabling visuomotor policies to achieve state-of-the-art dual-arm manipulation performance without sim-to-real transfer gaps.

What Problem Does This Paper Address?

Training visuomotor policies (e.g., Diffusion Policy) for complex, dual-arm robotic manipulation typically demands massive datasets of expert demonstrations. Today's practitioners face an agonizing trade-off:

  1. Human teleoperation / kinesthetic teaching produces physically grounded, high-fidelity data but is prohibitively slow and labor-intensive (taking ~90 seconds per demo).
  2. Simulation-based generation scales automatically but suffers from visual and dynamic sim-to-real gaps that cause failures in fine-grained, contact-rich tasks.
  3. Data augmentation via point-cloud warping creates severe visual distortion artifacts that degrade perceptual learning.

The central question is: How can we scale up real-world bimanual demonstrations without relying on simulators, manual replays, or visual distortions?

Methods

BiDemoSyn synthesizes data directly in the physical domain using a three-stage pipeline based on a single seed demonstration:

[1-Shot Kinesthetic Demo]
         │
         ▼
 1. Deconstruction ──► Breaks task into Invariant Blocks (e.g., pouring, twisting)
         │             & Variable Blocks (e.g., initial reaching/grasping).
         ▼
 2. Vision Alignment ─► VFM perception (Florence-2 + SAM2 / YOLOE) segments new 
         │              object instances; PCA on depth finds 6D poses to update grasps.
         ▼
 3. Trajectory Opt. ──► Solves IK reachability, verifies collision avoidance, and 
                        offsets endpoints by bounding box size (Δl, Δw, Δh).
         │
         ▼
[Scaled Real-World Dataset (D_syn)] ──► Train Bimanual Diffusion Policy (DP3 / EquiBot)
  • Semantic Deconstruction: Breaks the original trajectory into coarse blocks and minimal units called Atomic Execution Primitives (AEPs). Crucially, it categorizes them into invariant blocks (structural actions like twisting a cap or tilting a cup) and variable blocks (reaching and grasping, which depend strictly on initial object layout).
  • Vision-Guided Initial Frame Alignment: When new objects are placed on the table, vision foundation models (Florence-2 and SAM2) detect and segment the items. Using 3D image moments and PCA on depth point clouds, the system calculates the object’s 6D pose ($P_{\text{obj}} \in SE(3)$) and computes a rigid transformation to map the seed grasp pose to the novel scene.
  • Hierarchical Trajectory Optimization: Before accepting a synthetic path, inverse kinematics (IK) validates joint reachability and a collision checker ensures motion safety. Grasp offsets are scaled to adjust for differences in instance dimensions ($\Delta l, \Delta w, \Delta h$).
  • Operator UI Interface: An augmented-reality UI overlays digital grid targets on the live camera stream, letting a human quickly reposition objects while the system validates and saves feasible synthetic trajectories in roughly 5 seconds.

Datasets

The authors evaluated BiDemoSyn on a dual-arm real-robot testbed (two 6-DoF AUBO-i5 arms with parallel grippers and a Kingfisher stereo camera) across six contact-rich bimanual tasks, along with two extended long-horizon tasks:

  • plugpen: Aligning and inserting a pen body into a cap ($\pm$sub-millimeter tolerance).
  • inserting: 180° cup flip synchronized with vertical pen insertion.
  • unscrew: Left arm stabilizes bottle while right arm untwists the cap.
  • pouring: Tilting a water bottle into a mug held by the opposite arm.
  • pressing: Dispensing shampoo/water while catching liquid with a tilted cup.
  • reorient: Picking up a flat shovel/spoon, mid-air handover, and 180° flip.
  • Long-Horizon Additions: reorient + unscrew and unscrew + pouring.

Across all tasks, the framework synthesized between 1,000 and 7,700 demonstrations per task directly on the real setup.

Key Insights and Ideas

  • Functional Invariance vs. Contextual Adaptability: Complex manipulation doesn't require reinventing the entire motion for every scenario. Core physical interactions (e.g., the liquid-pouring arc or cap twisting) stay invariant; only the entry waypoints and spatial endpoints need geometric adaptation.
  • Eliminating "Sim-to-Real" by Staying in Reality: Rather than synthesizing pixels in simulation or warping 3D point clouds with artificial distortions, BiDemoSyn collects real RGB-D frames of real objects on the table and adjusts the trajectory analytically.
  • Behavioral Noise is Not Generalization: Recent work often adds random trajectory jittering to diversify data. The authors show that random low-level jitter often hurts policy convergence. Systematic spatial diversity (translating and rotating objects across a grid) combined with instance shape diversity is what actually drives out-of-distribution robustness.

Key Results

  • Synthesis Speed & Usability: BiDemoSyn generates a demonstration in ~5 seconds, vastly outpacing physical robot replay (YOTO at 42s) and manual teleoperation (91s), while avoiding the perspective distortions of 3D point-cloud editing (DemoGen).
  • Manipulation Success Rates: When training an equivariant diffusion policy (EquiBot):
  • In-Distribution (ID): BiDemoSyn achieved an average success rate of 86.7% across all six tasks, outperforming DemoGen (68.9%), YOTO (71.1%), and zero-shot baselines like ReKep (42.2%) and ODIL (54.4%).
  • Out-of-Distribution (OOD): On novel object instances and unseen positions, BiDemoSyn maintained 66.7% success, far ahead of DemoGen (40.6%) and YOTO (47.8%).
  • Data Scaling Laws: Performance scaled steadily with synthetic data volume (e.g., DP3 success increased from ~25% at 550 demos to >80% at 17,000 aggregate demos), proving that synthetic trajectories obey empirical imitation scaling laws.

Limitations and Future Work

  • Quasi-Static Assumption: Trajectory modulation assumes quasi-static conditions and cannot yet handle dynamic, ballistic actions (e.g., catching or dynamic throwing).
  • Deformable Objects & Complex Contacts: Tasks involving multi-stage interdependent friction and deformation (like tying shoelaces or folding flexible cloth) are currently out of scope.
  • Symmetric / Textureless Ambiguity: 47% of pipeline failures stem from orientation estimation errors (e.g., confusing concave vs. convex sides of identical reflective spoons) or pre-grasp collisions on irregular geometries.
  • Hardware Limitations: Binary grippers lacking tactile/torque feedback restrict fine-grained impedance control, which the authors aim to address in future work through adaptive contact modeling and dynamic visual servoing.

Summary updated

Abstract

Learning dexterous bimanual manipulation policies critically depends on large-scale, high-quality demonstrations, yet current paradigms face inherent trade-offs: teleoperation provides physically grounded data but is prohibitively labor-intensive, while simulation-based synthesis scales efficiently but suffers from sim-to-real gaps. We present BiDemoSyn, a framework that synthesizes contact-rich, physically feasible bimanual demonstrations from a single real-world example. The key idea is to decompose tasks into invariant coordination blocks and variable, object-dependent adjustments, then adapt them through vision-guided alignment and lightweight trajectory optimization. This enables the generation of thousands of diverse and feasible demonstrations within several hour, without repeated teleoperation or reliance on imperfect simulation. Across six dual-arm tasks, we show that policies trained on BiDemoSyn data generalize robustly to novel object poses and shapes, significantly outperforming recent baselines. By bridging the gap between efficiency and real-world fidelity, BiDemoSyn provides a scalable path toward practical imitation learning for complex bimanual manipulation without compromising physical grounding.
BibTeX
@misc{zhou2025one,
  author = {Huayi Zhou and Kui Jia},
  title = {{One-Shot Real-World Demonstration Synthesis for Scalable Bimanual Manipulation}},
  year = {2025},
  eprint = {2512.09297},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.09297},
}
Figure