Abstract
In this paper, we discuss a framework for teaching bimanual manipulation tasks by imitation. To this end, we present a system and algorithms for learning compliant and contact-rich robot behavior from human demonstrations. The presented system combines insights from admittance control and machine learning to extract control policies that can (a) recover from and adapt to a variety of disturbances in time and space, while also (b) effectively leveraging physical contact with the environment. We demonstrate the effectiveness of our approach using a real-world insertion task involving multiple simultaneous contacts between a manipulated object and insertion pegs. We also investigate efficient means of collecting training data for such bimanual settings. To this end, we conduct a human-subject study and analyze the effort and mental demand as reported by the users. Our experiments show that, while harder to provide, the additional force/torque information available in teleoperated demonstrations is crucial for phase estimation and task success. Ultimately, force/torque data substantially improves manipulation robustness, resulting in a 90% success rate in a multipoint insertion task. Code and videos can be found at https://bimanualmanipulation.com/
Introduction
Teaching robots complex, bimanual tasks through human demonstration remains a significant challenge, particularly in "contact-rich" environments. Real-world assembly, such as inserting a bracket onto four tight-fitting pins, requires more than just spatial mimicry. Most current imitation learning approaches focus on single-arm movements in unobstructed environments, failing to address two critical hurdles: coordination of internal forces between arms and temporal uncertainty. When two arms hold the same object, they can easily conflict, and any physical obstruction can desynchronize a robot's internal clock from its actual progress. This work introduces a framework that understands the relationship between force, space, and time to overcome these challenges.
Methodology
The proposed system utilizes a stacked control architecture designed for robustness and compliance. At its core is Ensemble Bayesian Interaction Primitives (EnBIP), which models tasks as joint probability distributions of spatial and temporal data. Unlike fixed-timer systems, EnBIP uses an Ensemble Kalman Filter (EnKF) for Dynamic Phase Estimation. By constantly updating its belief about task completion (0% to 100%) based on sensor readings, the robot can slow down, stop, or even reverse its progress when it encounters resistance.
Supporting this "brain" is an Admittance Control layer, acting as the system's "muscle." This layer makes the robot compliant, allowing it to behave like a spring-mass-damper system. This compliance is essential for "giving" slightly when encountering unexpected resistance, preventing hardware damage and facilitating smoother insertions.
Experimental Evaluation
The system was evaluated across three primary settings:
- Simulation: A MuJoCo environment featuring a UR5 robot performing two-pin insertions with tight tolerances (1mm and 5mm).
- Real-World Setup: A dual-robot station (UR5 and UR10) tasked with the four-pin bracket insertion.
- Human-Subject Study: A study involving 8 participants who provided 30 demonstrations each via two methods: kinesthetic teaching (direct physical manipulation) and Space-Mouse teleoperation (6-DoF joystick control).
Discussion and Key Insights
A central finding of this research is that temporal localization (phase) is as critical as spatial accuracy. In contact-rich tasks, the ability to realize that progress has stalled and adjust accordingly is the difference between success and failure.
Furthermore, the study highlighted a trade-off in data collection. While kinesthetic teaching is easier for humans, teleoperation provides significantly cleaner Force/Torque (F/T) data. In kinesthetic teaching, sensors are often "polluted" by the human's direct input on the robot arms. In teleoperation, the sensors exclusively capture the interaction between the robot and its environment. This high-quality force data proved to be the "secret sauce," increasing success rates from approximately 70% to 90%.
Results
The EnBIP system achieved a 90% success rate on the real-world bimanual insertion task, demonstrating significant robustness to varying starting positions. It substantially outperformed baselines such as Behavior Cloning (0% success) and Probabilistic Motor Primitives (ProMP) with Time Warping (20% success). Notably, the system showed remarkable robustness to disturbances; in simulations, the robot successfully completed tasks even when manually pushed backward, as the phase estimation correctly "rewound" task progress.
Conclusion and Future Work
While highly effective, the system currently relies on near-expert human demonstrations. Future work will explore methods to make the framework more robust to non-expert or "noisy" data, potentially by filtering out bad data or using reinforcement learning refinement. Nonetheless, the combination of Bayesian phase estimation and admittance control provides a powerful path forward for complex bimanual manipulation.
Summary updated