Multiagent Bidirectionally-Coordinated Nets for Learning to Play StarCraft Combat Games

Peng Peng, Quan Yuan, Ying Wen, Yaodong Yang, Zhenkun Tang, Haitao Long, Jun WangarXiv 2017 · OpenReview

Introduction

In the field of Artificial Intelligence, transitioning from single-agent systems to multiagent environments represents a significant leap in complexity. While single-agent tasks like chess or Go focus on individual decision-making, multiagent reinforcement learning (MARL) requires agents to coordinate their actions in dynamic, often adversarial settings. This paper introduces BiCNet (Bidirectionally-Coordinated Net), a framework designed to enable agents to learn sophisticated team-based tactics through emergent communication.

The Multiagent Challenge

MARL faces two primary hurdles: scalability and coordination. As the number of agents increases, the joint state-action space grows exponentially, making traditional centralized control methods computationally infeasible. Conversely, independent learners often fail to achieve global objectives because they ignore the actions of their teammates.

StarCraft micromanagement serves as an ideal "grand challenge" for MARL. Winning battles requires precise, frame-by-frame control of multiple units, each with unique attributes. The game's state space is vast, necessitating a solution that scales efficiently while fostering tight-knit coordination.

Methodology: The BiCNet Architecture

The core innovation of this work is the Multiagent Bidirectionally-Coordinated Network (BiCNet). Unlike previous approaches that rely on simple averaging or independent processing, BiCNet utilizes a bidirectional Recurrent Neural Network (Bi-RNN) as a communication layer.

The BiCNet architecture integrates a bidirectional RNN communication layer within an Actor-Critic framework to facilitate agent coordination.

Communication via Bi-RNN

In BiCNet, agents are conceptually arranged in a sequence. Information flows bidirectionally: from the first agent to the last and vice versa. This structure creates a latent communication channel where each agent's policy is informed by the states and potential actions of its peers. This sequential dependency allows agents to naturally adopt complementary roles, such as one unit drawing enemy fire while others execute a flanking maneuver.

Scalability and Learning

To address scalability, BiCNet employs parameter sharing among agents of the same type. This significantly reduces the number of trainable parameters and allows models trained on small groups to generalize to larger squads. The framework uses a vectorized Actor-Critic model where both the Actor (policy) and Critic (value function) are integrated with the Bi-RNN, allowing learning signals to propagate through the entire team.

Experimental Evaluation

The researchers evaluated BiCNet using the StarCraft engine across various combat scenarios, ranging from homogeneous battles (e.g., 15 Marines vs. 16 Marines) to heterogeneous encounters requiring different unit types to work together (e.g., Dropships and Tanks vs. Ultralisks).

Emergent Tactics

A key finding was the emergence of professional-level tactics without human supervision. Agents independently developed:

  • Focus Fire: Concentrating all attacks on a single enemy unit to eliminate it quickly.
  • Hit-and-Run: Attacking then immediately retreating to minimize damage taken, a tactic known as "kiting" in the gaming community.

Results and Performance

BiCNet consistently outperformed state-of-the-art baselines, including CommNet and GMEZO. Beyond StarCraft, a "Guessing Sum" toy game confirmed that the hidden states of the RNN carry meaningful, additive information, proving that agents pass messages to achieve collective goals.

Comparison of winning rates over training episodes shows BiCNet achieving higher performance and faster convergence than CommNet and GMEZO.

In a challenging "10 Marines vs. 13 Zerglings" scenario, BiCNet achieved a 64% win rate, compared to 44% for CommNet. Furthermore, BiCNet demonstrated superior learning efficiency; in a 5v5 Marine battle, it mastered focus-fire strategies in just 10 episodes, whereas competing models required significantly more training to reach lower proficiency levels.

Conclusion and Future Directions

BiCNet demonstrates that bidirectional communication layers can effectively solve complex coordination problems in MARL. By allowing agents to "talk" through latent hidden states, the model enables the discovery of sophisticated tactics that were previously thought to require human-designed heuristics.

Future work will focus on deciphering the "semantics" of the latent communication signals and exploring graph-based communication structures to remove the requirement for a fixed agent ordering. Testing these coordinated swarms against high-level human players remains the ultimate benchmark for this technology.

Summary updated

Abstract

Many artificial intelligence (AI) applications often require multiple intelligent agents to work in a collaborative effort. Efficient learning for intra-agent communication and coordination is an indispensable step towards general AI. In this paper, we take StarCraft combat game as a case study, where the task is to coordinate multiple agents as a team to defeat their enemies. To maintain a scalable yet effective communication protocol, we introduce a Multiagent Bidirectionally-Coordinated Network (BiCNet ['bIknet]) with a vectorised extension of actor-critic formulation. We show that BiCNet can handle different types of combats with arbitrary numbers of AI agents for both sides. Our analysis demonstrates that without any supervisions such as human demonstrations or labelled data, BiCNet could learn various types of advanced coordination strategies that have been commonly used by experienced game players. In our experiments, we evaluate our approach against multiple baselines under different scenarios; it shows state-of-the-art performance, and possesses potential values for large-scale real-world applications.
BibTeX
@misc{peng2017multiagent,
  author = {Peng Peng and Quan Yuan and Ying Wen and Yaodong Yang and Zhenkun Tang and Haitao Long and Jun Wang},
  title = {{Multiagent Bidirectionally-Coordinated Nets for Learning to Play StarCraft Combat Games}},
  year = {2017},
  eprint = {1703.10069},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1703.10069},
}
Figure