Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Real

Ofir Nachum, Michael Ahn, Hugo Ponte, Shixiang Gu, Vikash Kumar

TL;DR

This paper introduces "hierarchical sim2real," a novel reinforcement learning approach that enables mobile robots to perform complex manipulation tasks by combining locomotion and coordination. By breaking down tasks into hierarchical levels and leveraging domain randomization, the method achieves impressive zero-shot transfer from simulation to real-world robots, even for multi-agent scenarios.

What Problem Does This Paper Address?

Traditionally, manipulation and locomotion have been treated as separate problems in robotics. However, many real-world tasks, like a group of lions hunting prey or even humans walking, involve a seamless blend of both. Applying reinforcement learning (RL) to these complex, multi-agent manipulation-via-locomotion tasks is challenging due to the difficulty of modeling dynamics, the need for extensive real-world data, and the scalability issues of domain randomization. Existing RL approaches struggle to generalize to real-world scenarios, especially when dealing with multiple interacting agents and intricate behaviors. This paper aims to bridge this gap by developing a robust and scalable RL framework for these combined manipulation and locomotion tasks.

Methods

The core of the proposed method is hierarchical sim2real. This approach decomposes the complex task into two levels:

  • Low-level policy (π_low): This policy is trained to perform basic goal-reaching locomotion skills. It's trained in simulation using domain randomization to ensure robustness. The key idea here is that this policy learns to reach arbitrary (x, y) coordinates, effectively treating the robot as a mobile point.
  • High-level policy (π_high): This policy learns to solve the overall task by iteratively setting goals for the low-level policy. It operates at a higher level of abstraction, directing the low-level policy to achieve task objectives.

This hierarchical structure allows for modular domain randomization. Instead of applying extensive randomization to a single, monolithic policy, domain randomization is applied separately to each level. For the low-level policy, randomization focuses on dynamics (joint friction, mass, etc.) and environmental variations (height fields). For the high-level policy, randomization is simpler, often involving adding noise to the high-level actions to account for potential discrepancies between simulation and reality. This modularity makes the domain randomization process more tractable and scalable for complex tasks. The entire hierarchical policy is then transferred to the real world in a zero-shot manner.

Datasets

The paper does not explicitly mention or introduce new datasets. Instead, it relies on simulated environments (specifically, MuJoCo) for training and then evaluates the learned policies on real-world robotic platforms. The tasks themselves (Avoid, Push, Coordinate) serve as benchmarks for evaluating the proposed method.

Key Insights and Ideas

  • Hierarchy is Crucial for Sim2Real Scalability: The paper strongly suggests that a hierarchical approach is essential for making domain randomization effective and scalable for complex real-world tasks. Applying extensive randomization to a single policy can lead to overly conservative behaviors that fail in the real world.
  • Modularity in Domain Randomization: By separating the problem into low-level locomotion and high-level task planning, the required domain randomization for each level becomes simpler and more targeted, leading to better generalization.
  • Locomotion as a Primitive for Manipulation: The work demonstrates that robust, goal-conditioned locomotion skills can serve as powerful building blocks for more complex manipulation and coordination behaviors.
  • Zero-Shot Transfer is Achievable: The proposed hierarchical sim2real method enables successful transfer of learned policies to real-world robots without any fine-tuning, highlighting the effectiveness of their approach.
  • Height Field Randomization is Highly Impactful: The authors found that randomizing the terrain's height field was a particularly crucial randomization for achieving robust low-level locomotion policies.

Key Results

The paper evaluates its method on three tasks: Avoid, Push, and Coordinate, using a D'Kitty quadruped robot.

  • Real-World Performance: The hierarchical sim2real method achieved significantly higher success rates compared to non-hierarchical baselines across all tasks.
  • Avoid: 73.3% success (Hierarchical Sim2Real) vs. 0% (No hierarchy, with randomization) and 20% (No hierarchy, no randomization).
  • Push: 73.3% success (Hierarchical Sim2Real) vs. 0% (No hierarchy, with randomization) and 6.7% (No hierarchy, no randomization).
  • Coordinate: 90.0% success (Hierarchical Sim2Real) vs. 0% (No hierarchy, with randomization) and 30% (No hierarchy, no randomization).
  • Ablation Studies:
    • Non-hierarchical policies, even with domain randomization, performed poorly in the real world, suggesting that hierarchy is critical for sim2real transfer of complex behaviors.
  • The "No high-level randomization" ablation still performed well, indicating that the low-level policy's robustness is a strong foundation. However, adding high-level randomization further improved performance, reaching 90% on Coordinate.
  • Low-Level Policy Robustness: Table 2 shows that randomizations, especially height field randomization, are vital for the low-level policy to avoid falling in the real world, improving distance traveled by about 50%.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that even with their method, there can be a non-negligible gap between simulation and real-world behavior, leading to slight biases in movement.
  • While the method is generalizable, the current evaluation is focused on quadrupedal robots and specific manipulation tasks.
  • Future Work:
    • The authors intend to continue their work by further exploiting hierarchical structure to demonstrate more difficult tasks.
  • They also aim to explore scenarios with a greater number of cooperating agents.

Summary updated

Abstract

Manipulation and locomotion are closely related problems that are often studied in isolation. In this work, we study the problem of coordinating multiple mobile agents to exhibit manipulation behaviors using a reinforcement learning (RL) approach. Our method hinges on the use of hierarchical sim2real -- a simulated environment is used to learn low-level goal-reaching skills, which are then used as the action space for a high-level RL controller, also trained in simulation. The full hierarchical policy is then transferred to the real world in a zero-shot fashion. The application of domain randomization during training enables the learned behaviors to generalize to real-world settings, while the use of hierarchy provides a modular paradigm for learning and transferring increasingly complex behaviors. We evaluate our method on a number of real-world tasks, including coordinated object manipulation in a multi-agent setting. See videos at https://sites.google.com/view/manipulation-via-locomotion
BibTeX
@inproceedings{nachum2019multi,
  author = {Ofir Nachum and Michael Ahn and Hugo Ponte and Shixiang Gu and Vikash Kumar},
  title = {{Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Real}},
  booktitle = {Conference on Robot Learning},
  year = {2019},
  url = {https://openreview.net/forum?id=Dei_F3BEdsh},
}
Figure