Hierarchical Macro Strategy Model for MOBA Game AI

Bin Wu, Qiang Fu, Jing Liang, Peng Qu, Xiaoqian Li, Liang Wang, Wei Liu, Wei Yang, Yongsheng Liu

Introduction

Multiplayer Online Battle Arena (MOBA) games like Honour of Kings present a formidable challenge for Artificial Intelligence. Unlike Go, which has a state space of $10^{250}$, MOBA games reach complexities of $10^{20,000}$. They involve imperfect information due to the "fog of war," require seamless multi-agent coordination among five players, and suffer from sparse rewards where a single early-game error can lead to a loss much later.

While previous AI systems have excelled at "micro" tasks—such as precise spell-casting and combat mechanics—they often lack "macro" intelligence: the high-level strategic planning regarding map positioning and objective prioritization. Researchers from Tencent AI Lab have addressed this gap with the Hierarchical Macro Strategy (HMS) model.

Methods

The HMS model is a multi-task supervised learning framework designed to guide micro-level execution through high-level strategic layers.

Architecture of the Hierarchical Macro Strategy (HMS) model.

The architecture consists of two primary components:

  1. Phase Layer (The "When"): This layer identifies the current strategic stage of the game. Rather than using fixed time intervals, it predicts which major resources—such as specific turrets, the Dragon, or the Baron—are currently being contested.
  2. Attention Layer (The "Where"): This layer predicts the optimal region on the map for an agent. By treating the map as a grid and outputting an "attention" distribution, the model learns from pro-player data where the next point of engagement should be.
  3. Imitated Cross-agent Communication: To facilitate coordination without explicit communication logs, the model uses the predicted attention maps of teammates as input features. This allows an agent to "anticipate" and align its movements with its allies.

The macro-level outputs are integrated as embeddings into a micro-level model that manages frame-by-frame actions.

Datasets and Training

The model was trained on a massive dataset of approximately 300,000 professional-grade replays from the King Professional League (KPL) and top-tier "King" rank matches. This dataset encompasses 250 million frames. The input features include 85 visual features (represented as a 12x12 blurred map) and 181 attribute features, such as hero roles, gold, levels, and KDA statistics.

Key Insights and Ideas

A central innovation of this work is treating strategy as dynamic spatial attention. By predicting where a player should focus on the map, the AI can effectively bridge the gap between abstract goals and concrete movements.

Role-specific attention maps generated by the HMS model.

The model demonstrates that different hero roles require distinct strategic focuses. For instance, an Assassin like Hanxin has different spatial priorities compared to an Archer like Houyi. Furthermore, by tying game "phases" to specific map objectives, the AI gains measurable, concrete goals that mirror human strategic thinking.

Key Results

The HMS-powered AI was evaluated in 250 matches against human teams in the top 1% (King rank), achieving a 48% win rate. This performance indicates a near-professional level of play.

Ablation studies highlighted the importance of the hierarchical approach:

  • The HMS model defeated a version with only micro-level logic in 75% of matches.
  • The "Imitated Communication" mechanism improved the win rate from 37.5% to 62.5% against a non-communicating baseline.

Qualitatively, the AI exhibited sophisticated behaviors, such as automatically moving to protect high-value jungle zones at the start of a match, a hallmark of professional play.

Limitations and Future Work

While the macro strategy showed significant improvement, the AI's team-fighting coordination (micro) still lagged slightly behind the very best human teams. Future research may involve incorporating Monte Carlo Tree Search (MCTS) for proactive planning or adapting the "Phase + Attention" hierarchy to other complex genres like Real-Time Strategy (RTS) games.

Summary updated

Abstract

The next challenge of game AI lies in Real Time Strategy (RTS) games. RTS games provide partially observable gaming environments, where agents interact with one another in an action space much larger than that of GO. Mastering RTS games requires both strong macro strategies and delicate micro level execution. Recently, great progress has been made in micro level execution, while complete solutions for macro strategies are still lacking. In this paper, we propose a novel learning-based Hierarchical Macro Strategy model for mastering MOBA games, a sub-genre of RTS games. Trained by the Hierarchical Macro Strategy model, agents explicitly make macro strategy decisions and further guide their micro level execution. Moreover, each of the agents makes independent strategy decisions, while simultaneously communicating with the allies through leveraging a novel imitated cross-agent communication mechanism. We perform comprehensive evaluations on a popular 5v5 Multiplayer Online Battle Arena (MOBA) game. Our 5-AI team achieves a 48% winning rate against human player teams which are ranked top 1% in the player ranking system.
BibTeX
@inproceedings{wu2019hierarchical,
  author = {Bin Wu and Qiang Fu and Jing Liang and Peng Qu and Xiaoqian Li and Liang Wang and Wei Liu and Wei Yang and Yongsheng Liu},
  title = {{Hierarchical Macro Strategy Model for MOBA Game AI}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33011206},
}
Figure