Reflection of Episodes: Learning to Play Game from Expert and Self Experiences

Xiaojie Xu, Zongyuan Li, Chang Lu, Runnan Qi, Yanan Ni, Lumin Jiang, Xiangbei Liu, Xuebo Zhang, Yongchun Fang, Kuihua Huang, Xian Guo, Zhanghua Wu, Zhenya Li

TL;DR

This paper introduces the Reflection of Episodes (ROE) framework, a novel approach for Large Language Models (LLMs) to learn and improve their strategies in complex real-time strategy (RTS) games like StarCraft II. By leveraging both expert demonstrations and self-generated experiences, ROE enables LLMs to iteratively refine their decision-making, achieving impressive results against challenging AI opponents.

What Problem Does This Paper Address?

The paper tackles the challenge of enabling LLMs to learn and adapt effectively within complex, dynamic environments like StarCraft II. While LLMs excel at language tasks, their ability to strategize and improve over time in games with high-level decision-making, resource management, and combat is limited. Existing methods often focus on immediate actions without a robust mechanism for self-reflection and strategy iteration. The authors highlight that current LLM approaches in StarCraft II environments, while capable, tend to focus on immediate state details rather than overarching strategy and lack a reflective process for continuous improvement.

Methods

The core contribution is the Reflection of Episodes (ROE) framework, which comprises two main innovations:

  • Key Frame Selection based on Game Phase Division: To distill crucial information from the vast amount of game data (7000 frames per match), ROE employs a method that divides the game into distinct phases (e.g., early, mid, late game). Within each phase, it identifies and selects key frames that capture significant game events. This allows the LLM to process a more manageable and relevant subset of information for reflection.
  • Reflection Generation and Strategy Iteration: After a game concludes, the framework generates a "self-reflection" based on the selected key frames and the game's outcome. This self-reflection, combined with expert experience, forms a new prompt for the LLM. This iterative process allows the LLM to learn from its mistakes and successes, gradually improving its strategy over multiple game episodes. The process involves:
  • Expert Experience: Initial guidance from pre-defined expert strategies.
  • Self-Experience: Learning from the LLM's own game plays.
  • Key Frame Extraction: Identifying critical moments in the game.
  • Reflection Generation: The LLM analyzes key frames and its experience to generate insights.
  • Strategy Update: The generated reflection is incorporated into the LLM's prompt for the next game, leading to strategy iteration.

Datasets

The experiments were conducted within the TextStarCraft II environment, which provides an API for LLMs to interact with the game. No new datasets were introduced; rather, the framework was evaluated on its performance within this existing benchmark.

Key Insights and Ideas

  • The Power of Reflection: The paper strongly suggests that enabling LLMs to reflect on their past experiences, both successes and failures, is crucial for developing sophisticated strategies in complex environments.
  • Phase-Based Information Distillation: Simply processing all game data is inefficient. Dividing the game into phases and selecting key frames is an effective way to extract salient information for learning.
  • Bridging Expert and Self-Learning: Combining pre-existing expert knowledge with the LLM's own trial-and-error learning creates a powerful synergy for rapid improvement.
  • Iterative Strategy Refinement: The ROE framework demonstrates that continuous iteration, driven by reflection, can lead to significant strategic advancements, even against difficult AI opponents.

Key Results

The ROE framework demonstrated significant improvements in winning rates against the "Very Hard" difficulty AI in TextStarCraft II. Specifically, the method achieved a 20% winning rate against this challenging opponent, outperforming the baseline method which had a 0% win rate. The paper also shows comparative graphs illustrating that the ROE method leads to better resource management and unit production compared to the baseline, especially in the mid to late game. Ablation studies also confirmed the importance of both keyframe selection and the reflection mechanism for achieving these results.

Limitations and Future Work

The authors acknowledge that while their method shows promise, further exploration is needed. The paper mentions that the reflections obtained exhibit "partial strategy interpretability," suggesting that future work could focus on enhancing this interpretability. The framework's effectiveness in even more complex scenarios or against human players is also an area for future investigation. The paper also hints at further exploration of strategy generation approaches using LLMs.

Summary updated

Abstract

StarCraft II is a complex and dynamic real-time strategy (RTS) game environment, which is very suitable for artificial intelligence and reinforcement learning research. To address the problem of Large Language Model(LLM) learning in complex environments through self-reflection, we propose a Reflection of Episodes(ROE) framework based on expert experience and self-experience. This framework first obtains key information in the game through a keyframe selection method, then makes decisions based on expert experience and self-experience. After a game is completed, it reflects on the previous experience to obtain new self-experience. Finally, in the experiment, our method beat the robot under the Very Hard difficulty in TextStarCraft II. We analyze the data of the LLM in the process of the game in detail, verified its effectiveness.
BibTeX
@misc{xu2025reflection,
  author = {Xiaojie Xu and Zongyuan Li and Chang Lu and Runnan Qi and Yanan Ni and Lumin Jiang and Xiangbei Liu and Xuebo Zhang and Yongchun Fang and Kuihua Huang and Xian Guo and Zhanghua Wu and Zhenya Li},
  title = {{Reflection of Episodes: Learning to Play Game from Expert and Self Experiences}},
  year = {2025},
  eprint = {2502.13388},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.13388},
}
Figure