A Comparison of Imitation Learning Algorithms for Bimanual Manipulation

Michael Drolet, Simon Stepputtis, Siva Kailas, Ajinkya Jain, Jan Peters, Stefan Schaal, Heni Ben Amor

Abstract

Bimanual manipulation represents a significant challenge in robotics, requiring precise coordination and the ability to handle complex contact dynamics. While Imitation Learning (IL) offers a promising alternative to manually engineered reward functions in Reinforcement Learning, the relative performance of modern IL architectures remains under-explored in high-precision tasks. This paper benchmarks six prominent IL algorithms—Behavioral Cloning (BC), Action Chunking Transformer (ACT), Implicit Behavioral Cloning (IBC), Diffusion Policy, DAgger, and GAIL—on a bimanual peg-insertion task with a 1mm tolerance. Our findings indicate that action-chunking methods like ACT and Diffusion Policy provide superior robustness and efficiency, while adversarial methods like GAIL present significant computational and tuning challenges.

Introduction

Bimanual manipulation—the coordinated use of two robotic arms—is essential for tasks involving heavy loads or intricate assembly. However, the high-dimensional state space and complex interactions between arms make traditional programming and Reinforcement Learning (RL) difficult to implement. RL, in particular, often suffers from the "reward engineering" problem, where sub-optimal reward functions lead to unintended behaviors.

Imitation Learning (IL) circumvents these issues by learning directly from expert demonstrations. Despite its popularity, there is a lack of standardized benchmarking for IL in high-precision, contact-rich industrial scenarios. This study addresses this gap by evaluating a diverse set of IL algorithms on a bimanual task requiring sub-millimeter precision.

Methodology

Algorithms Evaluated

The study compares six distinct IL paradigms:

  1. Behavioral Cloning (BC): A standard supervised learning baseline mapping observations to actions.
  2. Action Chunking Transformer (ACT): A generative model that utilizes a transformer architecture to predict sequences (chunks) of future actions, mitigating the compounding error problem.
  3. Implicit Behavioral Cloning (IBC): An energy-based model that treats action selection as an optimization problem, seeking to minimize an energy landscape.
  4. Diffusion Policy: A method that generates actions via an iterative denoising process, similar to generative image models, allowing for complex multi-modal action distributions.
  5. DAgger: An interactive algorithm that queries an expert oracle to correct the robot's trajectory during rollouts.
  6. Generative Adversarial Imitation Learning (GAIL): An adversarial framework where a policy is trained to fool a discriminator that distinguishes between expert and agent behavior.

Experimental Setup

The benchmark was conducted in the MuJoCo physics engine using two UR5 robotic arms. The task involved a four-holed adapter transfer and its subsequent insertion onto four stationary pegs with a 1mm tolerance. To ensure focus on high-level coordination, the robots were controlled via an Operational Space Controller (OSC) in 3D task space.

Training data consisted of 200 expert demonstrations. To test robustness, the algorithms were evaluated across three noise regimes: Zero Noise, Low Noise, and High Noise, affecting both observations and action execution.

Results and Analysis

Performance and Robustness

ACT and Diffusion Policy emerged as the most capable algorithms, maintaining high success rates even in high-noise environments. Their success is largely attributed to "action chunking." By predicting a sequence of future actions based on a history of observations, these models better handle non-Markovian states—situations where the current sensor reading is insufficient to determine the next move, such as when a peg is momentarily stuck or misaligned.

In contrast, BC and IBC showed significant performance degradation when noise was introduced. While IBC is theoretically expressive, its sensitivity to hyperparameters and the complexity of the energy landscape made it less reliable in this high-precision context.

Computational Efficiency

A stark contrast was observed in training efficiency. GAIL required approximately 128,000 training rollouts (roughly 48 hours of compute) to achieve competitive performance. Furthermore, GAIL was highly sensitive to hyperparameter selection, making it the most difficult to deploy. BC and ACT were the most efficient, reaching peak performance in a fraction of the time required by adversarial or interactive methods.

Distributional Alignment vs. Success

The study noted that the Wasserstein distance—a metric used to measure how closely the agent's action distribution matches the expert's—did not always correlate with task success. This suggests that "looking" like an expert is insufficient; the ability to recover from small errors (robustness) is the more critical factor for high-precision manipulation.

Conclusion

This benchmark demonstrates that modern IL algorithms, particularly those utilizing action chunking and generative modeling like ACT and Diffusion Policy, are highly effective for complex bimanual tasks. These methods provide a balance of precision, robustness to noise, and training efficiency that surpasses traditional BC or adversarial approaches like GAIL.

Future research should address the limitations of this study, specifically the transition from simulation to real-world hardware (sim-to-real). Real-world variables such as friction, cable dynamics, and the inherent noise of human-provided demonstrations will provide the next rigorous test for these algorithms. Additionally, the inference latency of Diffusion and IBC models remains a consideration for high-speed robotic applications.

Summary updated

BibTeX
@article{drolet2024comparison,
  author = {Michael Drolet and Simon Stepputtis and Siva Kailas and Ajinkya Jain and Jan Peters and Stefan Schaal and Heni Ben Amor},
  title = {{A Comparison of Imitation Learning Algorithms for Bimanual Manipulation}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2024},
  url = {https://openreview.net/forum?id=qtz9MfL8sL},
}
Figure