Sequencing of multi-robot behaviors using reinforcement learning

Pietro Pierpaoli, Thinh T. Doan, Justin Romberg, Magnus EgerstedtControl 2021 · DOI

Introduction

Coordinating a team of robots to perform complex, multi-stage missions—such as convoy protection or object manipulation—remains a significant challenge in robotics. While individual "primitive" behaviors like formation control or leader-following are well-understood, the higher-level logic required to sequence these behaviors and tune their parameters in real-time is often computationally intractable. This paper proposes a hybrid framework that combines high-level Q-learning for behavior sequencing with online gradient descent for parameter optimization, enabling robot teams to autonomously learn and execute complex tasks in dynamic environments.

A Two-Layer Hybrid Framework

The core of the proposed approach is a decoupling of discrete sequencing logic from continuous parameter tuning. This prevents the "curse of dimensionality" typically associated with multi-robot learning.

High-Level Sequencing via Q-Learning

The system utilizes a library of pre-defined multi-robot behaviors (e.g., Static Formation, Cyclic Pursuit). A Q-learning agent operates at discrete "switching times," selecting the most appropriate behavior from the library based on the current environmental state.

Low-Level Parameter Tuning

Once a behavior is selected, it is not executed with static settings. Instead, the robots employ online gradient descent to continuously adjust internal parameters—denoted as $\theta$ (individual robot parameters) and $\phi$ (global mission parameters)—to minimize a local task performance cost.

Figure 5: Conceptual diagram of a multi-robot system with parameterized controllers (theta) and mission goals (phi), alongside the underlying communication graph.

The "Energy-Based" Interrupt Condition

A critical innovation is the trigger for transitioning between behaviors. Rather than switching at fixed intervals, the system monitors a Lyapunov-like "energy" function representing task error. When this energy falls below a predefined threshold ($\epsilon$), it indicates that the current behavior has reached a stable performance level, signaling the Q-learning agent to trigger the next transition.

Experimental Validation

The framework was evaluated using the Robotarium, a remote-access swarm robotics testbed. The researchers tested the algorithms on two primary mission scenarios:

  1. Convoy Protection: Protecting a moving asset through various environmental challenges.
  2. Object Manipulation: Coordinating multiple robots to locate and transport an object with unknown physical properties.

Performance was benchmarked against random behavior selection and "ideal" hand-coded solutions designed by human experts.

Key Insights and Results

The hybrid approach demonstrated significant advantages in both efficiency and robustness:

  • Emergent Strategy: In the object manipulation task, the robots successfully learned an optimal sequence: using "Leader-Follower" to locate the object, "Formation" to surround it, and "Cyclic Pursuit" for transport.
  • Superior Performance: In convoy protection, the RL-trained model matched the performance of expert-designed solutions. In object manipulation, the learned policy achieved approximately double the cumulative reward compared to random switching.
  • Adaptability: The system proved robust to environmental uncertainties, such as unknown mass distributions in the manipulation task, by adjusting parameters on the fly.

Conclusion and Future Work

By separating behavior selection from parameter optimization and using an energy-based trigger for transitions, this framework provides a scalable path for complex multi-robot coordination. Future research will focus on decentralizing the Q-learning process, moving away from a centralized supervisor toward fully distributed reinforcement learning. Additionally, exploring more sophisticated function approximators like Deep Q-Networks (DQN) could enhance the system's ability to handle continuous, high-dimensional state spaces.

Summary updated

Abstract

Abstract Given a collection of parameterized multi-robot controllers associated with individual behaviors designed for particular tasks, this paper considers the problem of how to sequence and instantiate the behaviors for the purpose of completing a more complex, overarching mission. In addition, uncertainties about the environment or even the mission specifications may require the robots to learn, in a cooperative manner, how best to sequence the behaviors. In this paper, we approach this problem by using reinforcement learning to approximate the solution to the computationally intractable sequencing problem, combined with an online gradient descent approach to selecting the individual behavior parameters, while the transitions among behaviors are triggered automatically when the behaviors have reached a desired performance level relative to a task performance cost. To illustrate the effectiveness of the proposed method, it is implemented on a team of differential-drive robots for solving two different missions, namely, convoy protection and object manipulation.
BibTeX
@inproceedings{pierpaoli2021sequencing,
  author = {Pietro Pierpaoli and Thinh T. Doan and Justin Romberg and Magnus Egerstedt},
  title = {{Sequencing of multi-robot behaviors using reinforcement learning}},
  booktitle = {Control},
  year = {2021},
  doi = {10.1007/s11768-021-00069-5},
}
Figure