Sub-optimal Policy Aided Multi-Agent Reinforcement Learning for Flocking Control

Yunbo Qiu, Yue Jin, Jian Wang, Xudong Zhang

TL;DR

This paper introduces SPA-MARL, a novel multi-agent reinforcement learning (MARL) algorithm that significantly boosts sample efficiency for flocking control. By leveraging a pre-existing, potentially sub-optimal policy (generated by Artificial Potential Fields), SPA-MARL allows agents to learn faster and achieve better performance than traditional MARL methods.

What Problem Does This Paper Address?

Flocking control, where multiple agents (like drones or robots) must move towards a target while maintaining formation and avoiding collisions, is a complex multi-agent problem. While Multi-Agent Reinforcement Learning (MARL) has shown promise, traditional MARL algorithms are notoriously data-hungry, requiring a vast number of interactions with the environment to learn effective policies. This inefficiency makes real-world deployment challenging. The paper aims to bridge this gap by improving the sample efficiency of MARL for flocking control.

Methods

The core innovation is the Sub-optimal Policy Aided Multi-Agent Reinforcement Learning (SPA-MARL) algorithm. It builds upon a standard actor-critic MARL framework (specifically, MADDPG) but introduces a novel "sub-optimal policy aiding" mechanism.

Here's how it works:

  • Sub-optimal Policy Generation: The paper uses a traditional Artificial Potential Field (APF) method to generate a "sub-optimal" policy. APF is a well-established technique for navigation and collision avoidance, but it can be rigid and may not achieve optimal performance.
  • Learning from the Sub-optimal Policy: During MARL training, each agent's learning policy is guided by this pre-computed sub-optimal policy. Instead of blindly imitating, SPA-MARL dynamically decides whether to learn from the sub-optimal policy's actions. It compares the estimated future rewards (Q-values) of its own proposed action versus the sub-optimal policy's action. If the sub-optimal policy's action is deemed better, the agent imitates it.
  • Dynamic Action Cloning Loss: To prevent the sub-optimal policy from hindering learning when the agent's own policy becomes superior, a "dynamic action cloning loss" is introduced. This loss selectively uses the sub-optimal policy's guidance only when its actions are evaluated as better by the critic.
  • Modified Critic Target: The critic's target value is also adjusted to consider the potential benefits of the sub-optimal policy, further aiding in learning more accurate value functions.

Datasets

The paper does not explicitly mention using pre-existing datasets. Instead, it describes building a custom simulation environment for flocking control. This environment is square, contains various obstacles (round and square), and initializes agents, targets, and obstacles randomly in each episode. The agents are round with a specific radius and mass. The target is square.

Key Insights and Ideas

  • Leveraging "Good Enough" Policies: The paper demonstrates that even a sub-optimal policy, if intelligently integrated, can significantly accelerate learning in MARL. This challenges the assumption that only expert demonstrations are valuable for imitation learning.
  • Dynamic Imitation: The core idea of dynamically deciding when to imitate based on a performance comparison (using Q-values) is crucial. This prevents the sub-optimal policy from becoming a ceiling on performance and allows the agent to surpass it.
  • Bridging Traditional Control and RL: SPA-MARL effectively bridges the gap between traditional control methods (like APF) and modern deep reinforcement learning, combining the strengths of both. APF provides a structured starting point, while RL offers adaptability and learning from experience.
  • Sample Efficiency is Key: The primary motivation and success of SPA-MARL lie in its dramatic improvement in sample efficiency, a critical bottleneck in MARL.

Key Results

  • Faster Training: SPA-MARL significantly speeds up the training process compared to standard MARL algorithms like MADDPG.
  • Improved Performance: The algorithm achieves higher success rates and rewards than both the baseline MARL (MADDPG) and the sub-optimal policy itself.
  • Outperforms Baselines: In statistical results, SPA-MARL shows superior success rates (0.890 vs. 0.857 for MADDPG) and rewards (1.031 vs. 0.823 for MADDPG). It also outperforms behavior cloning (BC) by a large margin.
  • Better Flock Control: Trajectory examples show SPA-MARL maintaining a tighter flock during navigation compared to MADDPG.
  • Ablation Studies: Experiments with ablations (SPA-MARL-actor and SPA-MARL-critic) confirm that both the actor and critic adaptations contribute to improved sample efficiency and performance, with the actor adaptation particularly helping in faster task completion.

Limitations and Future Work

  • Scalability: The authors acknowledge that studying the scalability of SPA-MARL with a larger number of agents is a direction for future work.
  • Alternative Sub-optimal Policies: The paper suggests exploring the use of other types of sub-optimal policies or different MARL algorithms within the SPA-MARL framework.
  • APF Limitations: While APF is used as a sub-optimal policy generator, its own limitations (e.g., potential for local minima, reliance on global information in traditional forms) are implicitly present. The paper addresses this by using a real-time, observation-based APF.

Summary updated

Abstract

Flocking control is a challenging problem, where multiple agents, such as drones or vehicles, need to reach a target position while maintaining the flock and avoiding collisions with obstacles and collisions among agents in the environment. Multi-agent reinforcement learning has achieved promising performance in flocking control. However, methods based on traditional reinforcement learning require a considerable number of interactions between agents and the environment. This paper proposes a sub-optimal policy aided multiagent reinforcement learning algorithm (SPA-MARL) to boost sample efficiency. SPA-MARL directly leverages a prior policy that can be manually designed or solved with a non-learning method to aid agents in learning, where the performance of the policy can be sub-optimal. SPA-MARL recognizes the difference in performance between the sub-optimal policy and itself, and then imitates the sub-optimal policy if the suboptimal policy is better. We leverage SPA-MARL to solve the flocking control problem. A traditional control method based on artificial potential fields is used to generate a sub-optimal policy. Experiments demonstrate that SPA-MARL can speed up the training process and outperform both the MARL baseline and the used sub-optimal policy.
BibTeX
@inproceedings{qiu2022sub,
  author = {Yunbo Qiu and Yue Jin and Jian Wang and Xudong Zhang},
  title = {{Sub-optimal Policy Aided Multi-Agent Reinforcement Learning for Flocking Control}},
  booktitle = {2022},
  year = {2022},
  doi = {10.1109/smc53654.2022.9945272},
}
Figure