Inverse Factorized Q-Learning for Cooperative Multi-agent Imitation Learning

The Viet Bui, Tien Mai, Thanh Hong NguyenarXiv 2023 · OpenReview

Abstract

This paper concerns imitation learning (IL) (i.e, the problem of learning to mimic expert behaviors from demonstrations) in cooperative multi-agent systems. The learning problem under consideration poses several challenges, characterized by high-dimensional state and action spaces and intricate inter-agent dependencies. In a single-agent setting, IL has proven to be done efficiently through an inverse soft-Q learning process given expert demonstrations. However, extending this framework to a multi-agent context introduces the need to simultaneously learn both local value functions to capture local observations and individual actions, and a joint value function for exploiting centralized learning. In this work, we introduce a novel multi-agent IL algorithm designed to address these challenges. Our approach enables the centralized learning by leveraging mixing networks to aggregate decentralized Q functions. A main advantage of this approach is that the weights of the mixing networks can be trained using information derived from global states. We further establish conditions for the mixing networks under which the multi-agent objective function exhibits convexity within the Q function space. We present extensive experiments conducted on some challenging competitive and cooperative multi-agent game environments, including an advanced version of the Star-Craft multi-agent challenge (i.e., SMACv2), which demonstrates the effectiveness of our proposed algorithm compared to existing state-of-the-art multi-agent IL algorithms.

Introduction

Imitation Learning (IL) is a cornerstone of modern AI, yet transitioning from single-agent to cooperative multi-agent systems introduces significant complexity. High-dimensional action spaces and the non-stationary nature of interacting agents make standard techniques brittle. Most state-of-the-art methods rely on adversarial training, which is prone to high-variance gradients and training instability.

To address these challenges, we propose Multi-agent Inverse Factorized Q-learning (MIFQ). This algorithm combines the stability of inverse soft-Q learning with the scalability of value-function factorization, providing a stable, scalable way for teams of agents to learn complex cooperative behaviors from expert demonstrations.

The MIFQ Framework

MIFQ rests on three main pillars to handle the scaling and stability problems inherent in multi-agent IL:

  1. Inverse Soft Q-Learning (IQ-Learn): Instead of an unstable generator-discriminator loop, MIFQ learns a single "soft" Q-function that implicitly represents both the reward and the policy, transforming a minimax problem into a stable minimization task.
  2. Value-Function Factorization: Borrowing from the QMIX architecture, each agent has a local Q-network. These local values are aggregated into a joint value by a Mixing Network.
  3. Hyper-networks: The weights of the Mixing Network are dynamically generated by Hyper-networks taking the global state as input, allowing the model to capture intricate inter-agent dependencies.
The architecture of MIFQ, featuring hyper-networks for mixing weights and decentralized agent networks.

A key theoretical contribution is the proof of convexity. If the mixing networks use non-negative weights and convex, non-decreasing activation functions (like ReLU), the multi-agent IL objective remains convex, ensuring stable optimization and convergence to a unique solution.

Experimental Evaluation

We evaluated MIFQ across three diverse environments:

  • SMACv2 (StarCraft Multi-Agent Challenge): A difficult benchmark with randomized team compositions and start positions.
  • Gold Miner: A competitive-cooperative 2D game requiring navigation and gold collection.
  • MPE (Multi-Particle Environments): A standard suite of coordination tasks such as simple_spread and speaker_listener.

Key Results and Analysis

MIFQ consistently outperformed existing state-of-the-art algorithms, including Behavioral Cloning (BC), Independent Inverse Q-learning (IIQ), and adversarial methods like MAGAIL.

Comparative performance of MIFQ against baseline methods across SMACv2, Gold Miner, and MPE environments.

As shown in the results table, MIFQ achieved a 63% win rate in the "Protoss 5vs5" task, significantly higher than MASQIL (46%) and MAGAIL (41%). Beyond raw performance, MIFQ demonstrated superior data efficiency, pulling away from competitors even with as few as 128 expert trajectories. The algorithm's adversarial-free nature resulted in smoother convergence curves and lower standard errors, indicating high training stability.

Conclusion

MIFQ provides a robust framework for multi-agent imitation learning by leveraging factorized Q-functions and ensuring objective convexity. While it significantly advances the state-of-the-art, future work will focus on closing the remaining performance gap with expert policies in the most complex scenarios and further scaling the architecture to systems with hundreds of agents.

Summary updated

Abstract

This paper concerns imitation learning (IL) (i.e, the problem of learning to mimic expert behaviors from demonstrations) in cooperative multi-agent systems. The learning problem under consideration poses several challenges, characterized by high-dimensional state and action spaces and intricate inter-agent dependencies. In a single-agent setting, IL has proven to be done efficiently through an inverse soft-Q learning process given expert demonstrations. However, extending this framework to a multi-agent context introduces the need to simultaneously learn both local value functions to capture local observations and individual actions, and a joint value function for exploiting centralized learning. In this work, we introduce a novel multi-agent IL algorithm designed to address these challenges. Our approach enables the centralized learning by leveraging mixing networks to aggregate decentralized Q functions. A main advantage of this approach is that the weights of the mixing networks can be trained using information derived from global states. We further establish conditions for the mixing networks under which the multi-agent objective function exhibits convexity within the Q function space. We present extensive experiments conducted on some challenging competitive and cooperative multi-agent game environments, including an advanced version of the Star-Craft multi-agent challenge (i.e., SMACv2), which demonstrates the effectiveness of our proposed algorithm compared to existing state-of-the-art multi-agent IL algorithms.
BibTeX
@misc{bui2023inverse,
  author = {The Viet Bui and Tien Mai and Thanh Hong Nguyen},
  title = {{Inverse Factorized Q-Learning for Cooperative Multi-agent Imitation Learning}},
  year = {2023},
  eprint = {2310.06801},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.06801},
}
Figure