Learning to Reason in Round-Based Games: Multi-Task Sequence Generation for Purchasing Decision Making in First-Person Shooters

Yilei Zeng, Deren Lei, Beichen Li, Gangrong Jiang, Emilio Ferrara, Michael Zyda

Abstract

Sequential reasoning is a complex human ability. While extensive research has focused on gaming AI in single continuous games, decision-making across a sequence of rounds remains less explored. Counter-Strike: Global Offensive (CS:GO) provides an ideal environment for studying multi-player round-based sequential reasoning. We propose a Sequence Reasoner with a Round Attribute Encoder and Multi-Task Decoder to interpret the strategies behind purchasing decisions. By formulating each round as a multi-task sequence generation problem and utilizing meta-learning (Reptile), our model learns to reason under complex economic constraints. Ablation studies and comparisons with a greedy approach demonstrate the effectiveness of our architecture in capturing long-term strategic purchasing behavior.

Introduction: The Challenge of Round-Based Reasoning

Most high-profile gaming AI research, such as OpenAI Five or AlphaStar, focuses on continuous real-time strategy. However, many competitive games are "round-based," where a match consists of discrete episodes. In CS:GO, players manage a "meta-economy" where decisions in one round (like an "eco round" to save money) directly impact future opportunities. Existing AI models often struggle with this because they treat games as single continuous tasks. This work aims to bridge that gap by teaching agents to reason across rounds, specifically focusing on the purchasing phase where players allocate limited resources.

Methodology: Multi-Task Sequence Reasoner

The core of our approach is treating weapon purchasing as a multi-task sequence generation problem, categorized into Guns, Grenades, and Equipment.

The proposed architecture featuring hierarchical encoders and multi-task decoders.

The architecture consists of several key components:

  • State Encoder: Utilizes hierarchical attention to represent the status of the player, teammates, and opponents.
  • Round Attribute Encoder (RAE): Acts as the model's memory, encoding history from previous rounds (purchases and performance) to adjust strategy based on match momentum.
  • Multi-Task Decoder with Gate Networks: Three specialized LSTM decoders generate sequences for different item categories. Gate Networks (binary classifiers) decide whether to initiate a sequence for a specific category (e.g., "Should I buy grenades?").
  • Meta-Learning (Reptile): To handle diverse player styles, we use the Reptile algorithm for few-shot learning, allowing the model to quickly adapt to specific player habits after observing only a few rounds.
  • Self-Critical Sequence Training (SCST): A reinforcement learning approach is used to optimize the model based on the F1 score of the final equipment set, mitigating exposure bias.

Dataset and Feature Engineering

We introduced a benchmark dataset derived from 5,167 professional CS:GO matches from 2019. Using a custom parser, we extracted "atomic actions" (individual purchases), player economy data, and scoreboard performance. This granular data allows the model to treat purchasing like generating a sentence, where items have a logical "purchasing habit" order (usually Guns → Grenades → Equipment).

Results and Discussion

Our model was evaluated against a "Greedy Algorithm" baseline and several ablated versions. The full architecture (Multi-Sequence + RAE + Gate) significantly outperformed the baseline.

Comparison of the proposed model against baselines and ablated versions.

Key findings include:

  • Overall Performance: The full model achieved an F1 score of 0.5670, more than doubling the performance of the Greedy baseline (0.2612).
  • Ablation Success: Removing the Gate Network or the RAE led to performance drops of 2-5%, validating that both multi-task structure and historical memory are essential.
  • Category Accuracy: The model excelled at predicting strategic items like grenades (F1 ~0.66) and equipment (F1 ~0.60).

Conclusion

This research demonstrates that multi-task sequence generation combined with meta-learning can effectively model complex, long-term economic strategies in round-based games. While a gap remains between AI and professional human performance—likely due to map-specific tactics and multi-agent coordination—this work opens doors for interpretable AI in episodic strategic decision-making.

Summary updated

Abstract

Sequential reasoning is a complex human ability, with extensive previous research focusing on gaming AI in a single continuous game, round-based decision makings extending to a sequence of games remain less explored. Counter-Strike: Global Offensive (CS:GO), as a round-based game with abundant expert demonstrations, provides an excellent environment for multi-player round-based sequential reasoning. In this work, we propose a Sequence Reasoner with Round Attribute Encoder and Multi-Task Decoder to interpret the strategies behind the round-based purchasing decisions. We adopt few-shot learning to sample multiple rounds in a match, and modified model agnostic meta-learning algorithm Reptile for the meta-learning loop. We formulate each round as a multi-task sequence generation problem. Our state representations combine action encoder, team encoder, player features, round attribute encoder, and economy encoders to help our agent learn to reason under this specific multi-player round-based scenario. A complete ablation study and comparison with the greedy approach certify the effectiveness of our model. Our research will open doors for interpretable AI for understanding episodic and long-term purchasing strategies beyond the gaming community.
BibTeX
@inproceedings{zeng2020learning,
  author = {Yilei Zeng and Deren Lei and Beichen Li and Gangrong Jiang and Emilio Ferrara and Michael Zyda},
  title = {{Learning to Reason in Round-Based Games: Multi-Task Sequence Generation for Purchasing Decision Making in First-Person Shooters}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aiide.v16i1.7446},
}
Figure