Mimicking To Dominate: Imitation Learning Strategies for Success in Multiagent Competitive Games

The Viet Bui, Tien Mai, Thanh Hong NguyenarXiv 2023 · OpenReview

Abstract

Training agents in multi-agent competitive games presents significant challenges due to their intricate nature. These challenges are exacerbated by dynamics influenced not only by the environment but also by opponents' strategies. Existing methods often struggle with slow convergence and instability. To address this, we harness the potential of imitation learning to comprehend and anticipate opponents' behavior, aiming to mitigate uncertainties with respect to the game dynamics. Our key contributions include: (i) a new multi-agent imitation learning model for predicting next moves of the opponents -- our model works with hidden opponents' actions and local observations; (ii) a new multi-agent reinforcement learning algorithm that combines our imitation learning model and policy training into one single training process; and (iii) extensive experiments in three challenging game environments, including an advanced version of the Star-Craft multi-agent challenge (i.e., SMACv2). Experimental results show that our approach achieves superior performance compared to existing state-of-the-art multi-agent RL algorithms.

Introduction

In the high-stakes world of multi-agent reinforcement learning (MARL), training agents to compete in complex, partially observable environments is a formidable task. The primary difficulty lies in the non-stationary nature of the environment; as agents learn, the optimal strategy shifts because opponents are also part of the dynamic. Most state-of-the-art methods, such as MAPPO or QMIX, focus on optimizing an agent's own policy or value function, often treating opponents as environmental noise. Furthermore, realistic games often feature partial observability, where agents can see an enemy's state but not their specific actions (e.g., commands or button presses). This lack of action visibility makes modeling and anticipating opponent behavior extremely difficult.

Methodology: The IMAX-PPO Framework

To overcome these challenges, we introduce IMAX-PPO (Imitation-enhanced Multi-Agent EXtended PPO). This algorithm integrates a state-only imitation learning (IL) component into the MAPPO framework to anticipate enemy moves.

Overview of the IMAX-PPO architecture, illustrating the integration of state-only imitation learning with the MAPPO framework.

The framework consists of two main components working in tandem:

  1. State-Only Imitation Learning: Drawing inspiration from IQ-Learn, we developed a multi-agent IL model that treats the opponent's next state as the target. Since opponent actions are hidden, the model learns to predict the enemy's future status (e.g., health, position) based on local observations.
  2. Policy Augmentation: These predictions are concatenated with the ally agent’s local observations. This "augmented input" provides a form of "future sight," allowing the PPO-based policy to make decisions based on where the enemy is predicted to be, rather than just where they are currently located.

The approach is theoretically grounded; we provide bounds on how changing ally policies affect the imitation learning outcome, ensuring stability throughout the training process. By framing imitation as a next-state prediction task, agents transition from being reactive to proactive.

Experimental Setup

We evaluated IMAX-PPO across three diverse and difficult benchmarks:

  • SMACv2 (StarCraft Multi-Agent Challenge v2): An evolution of the original SMAC featuring procedurally generated scenarios, randomized starting positions, and diverse team compositions to test generalization.
  • Google Research Football (GRF): A physics-based football simulator where we focused on "Academy" scenarios like 3-vs-1 and counter-attacks.
  • Gold Miner: A 2D competitive game where agents must out-mine heuristic-based enemies in a terrain filled with obstacles.

Results and Discussion

IMAX-PPO consistently outperformed state-of-the-art baselines across all tested environments, showing both higher win rates and faster convergence.

Training curves comparing IMAX-PPO against MAPPO and QMIX baselines across various SMACv2 scenarios.

In SMACv2, IMAX-PPO achieved significantly higher win rates. For instance, in "Protoss" scenarios, it reached an average win rate of 62.99%, compared to 53.56% for QMIX and 34.48% for MAPPO. Similar dominance was observed in Terran and Zerg maps. In Google Research Football, the results were even more pronounced; IMAX-PPO achieved a 98.71% average win rate across academy scenarios, while QMIX struggled significantly with a 9.11% average.

Beyond raw performance, IMAX-PPO demonstrated superior sample efficiency. As shown in the training curves, the algorithm often reached high win rates much earlier in the training process than its competitors. This suggests that understanding and anticipating the opponent provides a significant shortcut to developing effective strategies.

Conclusion

IMAX-PPO demonstrates that integrating imitation learning to predict opponent states can significantly enhance performance and stability in competitive MARL. While the current work assumes fixed enemy policies, future research will explore adaptive multi-agent IL to handle co-evolutionary settings where both sides learn and adapt simultaneously.

Summary updated

Abstract

Training agents in multi-agent competitive games presents significant challenges due to their intricate nature. These challenges are exacerbated by dynamics influenced not only by the environment but also by opponents' strategies. Existing methods often struggle with slow convergence and instability. To address this, we harness the potential of imitation learning to comprehend and anticipate opponents' behavior, aiming to mitigate uncertainties with respect to the game dynamics. Our key contributions include: (i) a new multi-agent imitation learning model for predicting next moves of the opponents -- our model works with hidden opponents' actions and local observations; (ii) a new multi-agent reinforcement learning algorithm that combines our imitation learning model and policy training into one single training process; and (iii) extensive experiments in three challenging game environments, including an advanced version of the Star-Craft multi-agent challenge (i.e., SMACv2). Experimental results show that our approach achieves superior performance compared to existing state-of-the-art multi-agent RL algorithms.
BibTeX
@misc{bui2023mimicking,
  author = {The Viet Bui and Tien Mai and Thanh Hong Nguyen},
  title = {{Mimicking To Dominate: Imitation Learning Strategies for Success in Multiagent Competitive Games}},
  year = {2023},
  eprint = {2308.10188},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.10188},
}
Figure