Playing Card-Based RTS Games with Deep Reinforcement Learning

Tianyu Liu, Zijie Zheng, Hongchang Li, Kaigui Bian, Lingyang SongIJCAI 2019 · OpenReview

Abstract: Game AI is of great importance as games are simulations of reality. Recent research on game AI has shown much progress in various kinds of games, such as console games, board games and MOBA games. However, the exploration in RTS games remains a challenge for their huge state space, imperfect information, sparse rewards and various strategies. Besides, the typical card-based RTS games have complex card features and are still lacking solutions. We present a deep model SEAT (selection-attention) to play card-based RTS games. The SEAT model includes two parts, a selection part for card choice and an attention part for card usage, and it learns from scratch via deep reinforcement learning. Empirical results show that the SEAT model agent makes it to reach a high winning rate against rule-based agents and decision-tree-based agent.

Introduction

Real-Time Strategy (RTS) games represent a significant frontier for Artificial Intelligence. Unlike board games with discrete states, RTS games like Clash Royale present massive state spaces, imperfect information, and the need for high-precision spatial decision-making. Card-based RTS games add further complexity through unique card attributes (HP, damage, abilities) and the critical management of resources like "elixir."

Existing heuristic methods often fail to capture the nuances of defensive "pulling" or aggressive "bridge spam" tactics. To address this, researchers have developed SEAT (Selection-Attention), a Deep Reinforcement Learning (DRL) framework that masters these complexities by decoupling the decision process.

The SEAT Model Architecture

The core innovation of the SEAT model is its "divide and conquer" approach to the action space. Instead of learning a single mapping from state to every possible card-coordinate pair, the model splits the task into two specialized components:

  1. The Selection Part: This component determines which card to play. It processes the battlefield state—represented as a multi-channel grid of unit health, class, and faction—alongside specific card features using a combination of Convolutional Neural Networks (CNN) and Multi-Layer Perceptrons (MLP).
  2. The Attention Part: Once a card is chosen, this component determines where to place it. By generating an "attention map" over the battlefield, the model identifies optimal deployment coordinates based on enemy concentrations and tower health.
The SEAT model architecture, decoupling card selection from spatial deployment.

Experimental Setup and Training

The model was trained from scratch in a custom-built Clash Royale simulation. A key training strategy involved the "abandoning-rate" ($\epsilon$). By encouraging the model to "do nothing" 50% of the time during early training, the agent learned the value of patience and elixir conservation—a critical skill for high-level play.

The evaluation utilized a standardized deck of eight cards, including multi-unit troops (Ninja, BatSwarm), tower-targeted units (ApeWarior), and area-of-effect spells (FireBall, Zap). Performance was measured against three benchmark agents: Aggressive Rule-Based (ARB), Defensive Rule-Based (DRB), and a sophisticated Decision-Tree (DT) agent.

Results and Tactical Emergence

The SEAT-agent demonstrated overwhelming superiority over traditional AI approaches. It achieved a 90% win rate against both aggressive and defensive rule-based agents. Even against the more complex Decision-Tree agent, which combines multiple strategies, SEAT maintained a robust 70% win rate.

Winning rates of the SEAT-agent against rule-based and decision-tree agents.

Beyond raw statistics, the model exhibited emergent tactical behavior. It learned to use low-cost units to distract high-damage enemies and timed its spells to maximize area-of-effect damage, effectively mimicking "pro" human maneuvers without explicit instruction.

Conclusion

The SEAT model provides a scalable solution for the complex action spaces of card-based RTS games. While currently limited to fixed decks and simulated opponents, the success of the selection-attention decoupling suggests a path forward for more generalized game AI. Future work will focus on expanding the model's capability to handle dynamic deck building and testing against high-level human players.

Summary updated

Abstract

Game AI is of great importance as games are simulations of reality. Recent research on game AI has shown much progress in various kinds of games, such as console games, board games and MOBA games. However, the exploration in RTS games remains a challenge for their huge state space, imperfect information, sparse rewards and various strategies. Besides, the typical card-based RTS games have complex card features and are still lacking solutions. We present a deep model SEAT (selection-attention) to play card-based RTS games. The SEAT model includes two parts, a selection part for card choice and an attention part for card usage, and it learns from scratch via deep reinforcement learning. Comprehensive experiments are performed on Clash Royale, a popular mobile card-based RTS game. Empirical results show that the SEAT model agent makes it to reach a high winning rate against rule-based agents and decision-tree-based agent.
BibTeX
@inproceedings{liu2019playing,
  author = {Tianyu Liu and Zijie Zheng and Hongchang Li and Kaigui Bian and Lingyang Song},
  title = {{Playing Card-Based RTS Games with Deep Reinforcement Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=v9EdZBgidD},
}
Figure