Learning multi-robot coordination from demonstrations

Xuan Wang, Wanxin Jin

Abstract

This paper develops a Distributed Differentiable Dynamic Game (D3G) framework, which enables learning multi-robot coordination from demonstrations. We represent multi-robot coordination as a dynamic game, where the behavior of a robot is dictated by its own dynamics and objective that also depends on others' behavior. The coordination thus can be adapted by tuning the objective and dynamics of each robot. The proposed D3G enables each robot to automatically tune its individual dynamics and objectives in a distributed manner by minimizing the mismatch between its trajectory and demonstrations. This learning framework features a new design, including a forward-pass, where all robots collaboratively seek Nash equilibrium of a game, and a backward-pass, where gradients are propagated via the communication graph. We test the D3G in simulation with two types of robots given different task configurations. The results validate the capability of D3G for learning multi-robot coordination from demonstrations.

Introduction

Coordinating a large fleet of robots—whether they are drones forming a light show or warehouse bots moving pallets—is notoriously difficult. Traditionally, engineers have to manually design "objective functions" (the mathematical rules for what a robot wants to achieve) through tedious trial and error. A more intuitive approach is Learning from Demonstrations (LfD), where robots watch an expert and infer the underlying rules.

However, when multiple robots interact, they are essentially playing a "dynamic game" where every robot's move depends on everyone else's. Solving this "Inverse Dynamic Game" (IDG) is computationally expensive. Existing methods usually rely on a central computer to process everything, which creates a massive bottleneck: as you add more robots, the math becomes exponentially harder, eventually crashing the system. This paper introduces D3G to address these scalability and complexity issues.

The D3G Framework

The Distributed Differentiable Dynamic Game (D3G) framework treats coordination as a Nash Equilibrium—a state where no robot can improve its own performance by changing its path, given what its neighbors are doing. The framework operates through a bi-level optimization process:

  1. Forward Pass: Robots collaboratively seek the Nash equilibrium of the game based on their current local objective functions and dynamics.
  2. Backward Pass (Inverse Learning): The system adjusts those rules so the predicted movement matches the expert's demonstration.
Overview of the D3G framework, illustrating the interaction between local parameterized optimal control, Nash equilibrium solving, and distributed inverse learning via differentiable KKT conditions.

The core technical innovation is the Distributed Diff-KKT Solver. To update the rules, the system needs to know how a small change in a robot's parameters affects the entire group's equilibrium. Usually, this requires a massive, centralized matrix calculation. D3G breaks this down into a distributed solver where each robot only performs local calculations and shares small updates with its immediate neighbors. This allows for local gradient descent, which is faster and maintains stability across the fleet.

Experimental Setup

The authors generated expert demonstrations within the Gazebo simulation environment using ROS (Robot Operating System). The demonstrations consisted of human-induced or noise-injected optimal trajectories for specific tasks. The experiments utilized heterogeneous fleets consisting of TurtleBot3 Burger and Waffle models, which have different physical sizes, weights, and movement constraints, testing the framework's ability to handle diverse robot dynamics.

Key Results and Discussion

The D3G framework demonstrated significant improvements in computational efficiency and generalization over existing methods:

  • Scalability: In a swapping task, as the number of robots increased from 4 to 20, D3G's computation time remained relatively flat. In contrast, centralized methods like GT-IRL and IKKT saw their computation times skyrocket; IKKT was nearly four times slower than D3G at the 20-robot mark.
  • Zero-Shot Generalization: After learning a swapping task with 6 robots, the system successfully coordinated 16 robots in a much more crowded space without any additional training. This is possible because the robots learn objective functions (e.g., "maintain distance") rather than specific paths.
  • Complex Coordination: The system successfully learned cooperative payload transport and formation maintenance through obstacles, intelligently deforming shapes only when necessary to avoid collisions.
Visualized trajectories of a multi-robot fleet (T=12) demonstrating coordinated movement learned through the D3G framework.

Conclusion and Future Work

D3G represents a major step forward in decentralized multi-robot learning. By leveraging the differentiability of KKT conditions, the sensitivity of the entire system can be solved through local consensus. Future work will focus on addressing real-world communication challenges like packet loss and delays, as well as transitioning from simulation to physical hardware. Additionally, the authors suggest evolving this into a reinforcement learning paradigm where robots learn coordination rules through self-exploration rather than relying solely on expert demonstrations.

Summary updated

Abstract

This paper develops a Distributed Differentiable Dynamic Game (D3G) framework, which enables learning multi-robot coordination from demonstrations. We represent multi-robot coordination as a dynamic game, where the behavior of a robot is dictated by its own dynamics and objective that also depends on others' behavior. The coordination thus can be adapted by tuning the objective and dynamics of each robot. The proposed D3G enables each robot to automatically tune its individual dynamics and objectives in a distributed manner by minimizing the mismatch between its trajectory and demonstrations. This learning framework features a new design, including a forward-pass, where all robots collaboratively seek Nash equilibrium of a game, and a backward-pass, where gradients are propagated via the communication graph. We test the D3G in simulation with two types of robots given different task configurations. The results validate the capability of D3G for learning multi-robot coordination from demonstrations.
BibTeX
@misc{wang2022learning,
  author = {Xuan Wang and Wanxin Jin},
  title = {{Learning multi-robot coordination from demonstrations}},
  year = {2022},
  eprint = {2207.08892},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2207.08892},
}
Figure