M2RL: A Multi-player Multi-agent Reinforcement Learning Framework for Complex Games

Tongtong Yu, Chenghua He, Qiyue YinIJCAI 2024

Introduction

Training artificial intelligence for complex, large-scale games like StarCraft II, Dota 2, or massively multiplayer online (MMO) environments represents a significant leap in difficulty compared to classic "toy" environments like Atari. Researchers in this field face a "four-headed monster" of challenges that hinder the development of robust, professional-level agents:

  1. Heterogeneous Agents: Units or players often possess vastly different roles, abilities, and objectives, requiring specialized logic rather than a one-size-fits-all approach.
  2. Asynchronicity: Agents do not always act simultaneously, and the environment may provide delayed or sparse feedback, complicating the credit assignment problem.
  3. Population Dynamics: To achieve robustness, an AI must train against a diverse and evolving set of opponents (population-based training) rather than a single static version of itself.
  4. Hardware Scaling: Coordinating training across dozens of CPUs and GPUs while maintaining high data throughput is a massive engineering hurdle.

While existing frameworks like SeedRL or MAVA address specific facets of these problems, they often lack a unified approach that handles multi-agent decision engineering, population evolution, and distributed training simultaneously. M2RL (Multi-player Multi-agent Reinforcement Learning) is an open-source distributed framework designed to bridge this gap.

The M2RL Architecture

The core of M2RL is its modular Actor-Learner-Buffer-Evaluator-Server architecture. This design decouples the various stages of the reinforcement learning pipeline to maximize efficiency and flexibility.

The M2RL distributed architecture featuring the Server, Task-specific Evaluators, Actors, Learners, and Buffers.

In this framework, the Server acts as the central orchestrator, managing player populations and dispatching parameters. Each Task (representing a specific player or faction) contains an Evaluator that coordinates local training. The Actors interact with the environment (Env) to collect samples, which are stored in a Buffer. The Learner then pulls data from the buffer to update the neural network models. By building on top of the Ray distributed computing library, M2RL optimizes resource usage: neural network inference and optimization are offloaded to GPUs, while environment interaction and sampling remain on CPUs.

Decision Engineering and Population Evolution

M2RL's novelty lies in its integration of high-level strategy with low-level engineering. The framework identifies "Three Pillars" necessary for success in complex environments:

  • Multi-agent Decision Engineering: M2RL provides specialized modules for State Engineering (transforming raw game data into usable features), Action Engineering (handling complex action spaces, including "momentary" vs. "continuous" actions), and Reward Engineering (using reward shaping and intrinsic rewards to mitigate the sparse reward problem).
  • Population Evolution: Training is treated as an iterative process where players are wrapped as independent tasks. This allows for asymmetrical training—for instance, a Red team and a Blue team can follow different evolutionary strategies, such as self-play or fictitious self-play, to ensure the resulting agents are robust against various playstyles.
  • Asymmetry as a First-Class Citizen: Unlike many frameworks that assume symmetry between players, M2RL acknowledges that real-world games often involve factions with different goals and capabilities. It allows for independent evolutionary paths for different " factions" within the same training run.

Experimental Validation and Results

The effectiveness of M2RL was validated in two highly complex game environments: Neural MMO, a 16-player massively multi-agent survival game, and Wargame, a two-player strategic game with unbalanced abilities.

Performance results showing M2RL's iterative improvement (a) and the dynamics of population evolution (b).

In the IJCAI 2022 Neural MMO Challenge, bots trained with M2RL secured a silver award. As shown in Figure 6(a), the agents demonstrated clear iterative improvement over successive generations, eventually surpassing baseline AI models and professional-level demonstrations. Figure 6(b) illustrates the population dynamics, where different strategies compete and evolve, leading to a more stable and dominant final agent.

In the Wargame scenario, the framework successfully trained bots capable of defeating professional-level AI. The results highlighted that the strategies became measurably stronger and more stable through the population evolution process. Furthermore, the system demonstrated high data throughput, efficiently utilizing 4x RTX 2080 Ti GPUs and 96 CPU cores, proving its scalability on mid-tier server hardware.

Conclusion

M2RL provides a comprehensive, modular solution for the challenges of multi-player, multi-agent reinforcement learning. By unifying decision engineering and population evolution within a scalable distributed architecture, it lowers the barrier for researchers tackling complex game environments. While currently focused on architectural efficiency, future work aims to expand the library of pre-built decision engineering templates, making the framework even more accessible for new and diverse applications.

Summary updated

BibTeX
@inproceedings{yu2024m,
  author = {Tongtong Yu and Chenghua He and Qiyue Yin},
  title = {{M2RL: A Multi-player Multi-agent Reinforcement Learning Framework for Complex Games}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure