On Efficient Reinforcement Learning for Full-length Game of StarCraft II

Ruo-Ze Liu, Zhen-Jia Pang, Zhou-Yu Meng, Wenhai Wang, Yang Yu, Tong Lu

TL;DR

This paper tackles the immense challenge of training reinforcement learning agents for the full-length game of StarCraft II, a notoriously complex real-time strategy game. They introduce a hierarchical reinforcement learning architecture combined with macro-actions derived from expert demonstrations, achieving impressive win rates against increasingly difficult AI opponents, even with limited computational resources.

What Problem Does This Paper Address?

StarCraft II (SC2) presents a grand challenge for reinforcement learning (RL) due to its massive state and action spaces, long time horizons, and imperfect information. Traditional RL methods struggle with such complexity. While prior work has explored parts of SC2 or used simplified versions, achieving strong performance in the full-length game, especially against sophisticated AI, remains a significant hurdle. This paper aims to develop an efficient RL approach that can learn to play SC2 effectively without requiring exorbitant computational resources or extensive human domain knowledge.

Methods

The core of this work lies in a hierarchical reinforcement learning (HRL) framework. This architecture decomposes the complex SC2 problem into manageable sub-problems.

  • Hierarchical Architecture: A two-layer (and later three-layer) hierarchy is employed. A controller makes high-level decisions at a slower timescale, selecting a sub-policy (e.g., for base management or combat). These sub-policies then execute actions at a faster timescale. This modularity allows for easier training and scaling.
  • Macro-Actions: To address the huge action space, the authors use data mining (PrefixSpan) on expert replays to extract frequently occurring sequences of actions. These sequences are then treated as macro-actions, significantly reducing the effective action space for the RL agent.
  • Curriculum Learning: Agents are trained progressively, starting from easier AI difficulty levels and gradually transferring to harder ones. This helps stabilize training and improve performance on more challenging opponents.
  • Reward Design: While exploring various reward functions (win/loss, Blizzard score), they found that a carefully designed reward function, combining dense and sparse rewards, was crucial for sub-policies. For higher difficulty levels, they found that using only the win/loss reward was effective.
  • Training Algorithm: They utilize the Proximal Policy Optimization (PPO) algorithm, with modifications like entropy bonus for exploration. They also investigate strategies like alternate updating of controller and sub-policies to improve stability.

Datasets

  • Expert Replays: The primary data source for generating macro-actions and for supervised learning (in the mini-AlphaStar comparison) are expert replays from StarCraft II.
  • SC2LE (StarCraft II Learning Environment): This is the environment used for training and evaluation. The paper uses a 64x64 map called Simple64.

Key Insights and Ideas

  • Hierarchy is Key for Complexity: The hierarchical approach effectively breaks down the massive state and action spaces of SC2, making it tractable for RL.
  • Macro-Actions Boost Efficiency: Mining macro-actions from expert data significantly reduces the action space, leading to faster learning and improved performance, especially when combined with RL.
  • Curriculum Learning is Crucial for Difficulty Progression: Training agents from easy to hard levels is essential for stable learning and achieving high win rates against increasingly difficult AI.
  • Limited Resources, Strong Performance: The paper demonstrates that effective SC2 agents can be trained with significantly fewer computational resources than state-of-the-art systems like AlphaStar, thanks to their hierarchical approach and macro-action strategy.
  • Win/Loss Reward Can Be Sufficient: While dense rewards can aid learning, the paper shows that for high-difficulty, cheating AI, a sparse win/loss reward can be sufficient and even preferable for training.

Key Results

  • Against Level-1 AI (restrictive units, 64x64 map): Achieved a 99% win rate.
  • Against Level-7 AI (most difficult non-cheating): Achieved a 93% win rate.
  • Against Cheating Levels (Level-8, -9, -10): With a final 3-layer hierarchical architecture and specific training tricks, they achieved win rates of 96%, 97%, and 94%, respectively.
  • Comparison with mini-AlphaStar (mAS): Their hierarchical approach was shown to be more effective and efficient when using limited computational resources compared to a scaled-down version of AlphaStar.
  • Scalability: The method showed effectiveness on different maps (Flat64, Simple96), demonstrating its extensibility.

Limitations and Future Work

  • Map Size and Unit Variety: The current experiments are limited to a 64x64 map and only two combat unit types. The authors acknowledge that larger maps and more unit variety would present greater challenges.
  • Human Knowledge: While aiming to reduce human knowledge, some aspects like reward design and macro-action generation still rely on expert data.
  • Future Work: The authors suggest exploring more efficient RL algorithms for complex games like StarCraft II, and continuing to address the challenge of requiring less human knowledge. They also mention the potential for further hierarchical extensions.

Summary updated

Abstract

StarCraft II (SC2) poses a grand challenge for reinforcement learning (RL), of which the main difficulties include huge state space, varying action space, and a long time horizon. In this work, we investigate a set of RL techniques for the full-length game of StarCraft II. We investigate a hierarchical RL approach involving extracted macro-actions and a hierarchical architecture of neural networks. We investigate a curriculum transfer training procedure and train the agent on a single machine with 4 GPUs and 48 CPU threads. On a 64x64 map and using restrictive units, we achieve a win rate of 99% against the level-1 built-in AI. Through the curriculum transfer learning algorithm and a mixture of combat models, we achieve a 93% win rate against the most difficult non-cheating level built-in AI (level-7). In this extended version of the paper, we improve our architecture to train the agent against the cheating level AIs and achieve the win rate against the level-8, level-9, and level-10 AIs as 96%, 97%, and 94%, respectively. Our codes are at https://github.com/liuruoze/HierNet-SC2. To provide a baseline referring the AlphaStar for our work as well as the research and open-source community, we reproduce a scaled-down version of it, mini-AlphaStar (mAS). The latest version of mAS is 1.07, which can be trained on the raw action space which has 564 actions. It is designed to run training on a single common machine, by making the hyper-parameters adjustable. We then compare our work with mAS using the same resources and show that our method is more effective. The codes of mini-AlphaStar are at https://github.com/liuruoze/mini-AlphaStar. We hope our study could shed some light on the future research of efficient reinforcement learning on SC2 and other large-scale games.
BibTeX
@article{liu2022efficient,
  author = {Ruo-Ze Liu and Zhen-Jia Pang and Zhou-Yu Meng and Wenhai Wang and Yang Yu and Tong Lu},
  title = {{On Efficient Reinforcement Learning for Full-length Game of StarCraft II}},
  journal = {J. Artif. Intell. Res.},
  year = {2022},
  url = {https://openreview.net/forum?id=kAjhd15RlP},
}
Figure