SPRING: Studying the Paper and Reasoning to Play Games

Yue Wu, So-Yeon Min, Shrimai Prabhumoye, Yonatan Bisk, Ruslan Salakhutdinov, Amos Azaria, Tom M. Mitchell, Yuanzhi Li

Abstract

Open-world survival games pose significant challenges for AI algorithms due to their multi-tasking, deep exploration, and goal prioritization requirements. Despite reinforcement learning (RL) being popular for solving games, its high sample complexity limits its effectiveness in complex open-world games like Crafter or Minecraft. We propose a novel approach, SPRING, to read the game's original academic paper and use the knowledge learned to reason and play the game through a large language model (LLM). Prompted with the LaTeX source as game context and a description of the agent's current observation, our SPRING framework employs a directed acyclic graph (DAG) with game-related questions as nodes and dependencies as edges. We identify the optimal action to take in the environment by traversing the DAG and calculating LLM responses for each node in topological order, with the LLM's answer to final node directly translating to environment actions. In our experiments, we study the quality of in-context "reasoning" induced by different forms of prompts under the setting of the Crafter open-world environment. Our experiments suggest that LLMs, when prompted with consistent chain-of-thought, have great potential in completing sophisticated high-level trajectories. Quantitatively, SPRING with GPT-4 outperforms all state-of-the-art RL baselines, trained for 1M steps, without any training. Finally, we show the potential of games as a test bed for LLMs.

Introduction

Traditional Reinforcement Learning (RL) agents typically master complex video games by playing millions of matches, slowly learning through trial and error. However, this approach suffers from high sample complexity and struggles to incorporate prior knowledge. Humans, by contrast, do not start a game by walking into walls for hours; we read instructions, watch tutorials, or study mechanics.

Open-world survival games like Crafter are particularly difficult for AI because they require multi-tasking, long-term planning, and the ability to prioritize conflicting goals (e.g., "should I find food or mine iron?"). The SPRING framework addresses this by allowing an agent to "read the manual"—specifically the original academic paper of the game—to achieve high-level performance immediately without environment-specific training.

The SPRING Framework

The SPRING framework operates in two distinct stages: Studying the Paper and Reasoning to Act.

The SPRING framework: An LLM studies the game's research paper to extract context, then uses a QA-DAG to reason through gameplay observations.

Phase 1: Studying the Paper

The agent is provided with the LaTeX source code of the game's original research paper. An LLM filters the paper paragraph-by-paragraph to identify relevant mechanics and strategies. This process involves identifying paragraphs containing game mechanics, extracting relevant information through question-answering, and removing duplicate items across sections. The result is a concise "context string" that describes the game's tech tree, objectives, and survival requirements.

The extraction pipeline: Filtering paragraphs, extracting information via QA, and concatenating relevant context.

Phase 2: Reasoning via QA-DAG

To ensure consistent gameplay, SPRING utilizes a Directed Acyclic Graph (DAG) of questions. Rather than asking the LLM for a single action, the agent answers a sequence of dependent questions, such as "What objects do I see?", "What was my last action?", and "What are the requirements for my current sub-task?". By traversing this graph, the LLM builds a logical chain of thought. The answer to the final node is then mapped to one of the game's discrete actions.

Results and Discussion

SPRING's performance in the Crafter environment demonstrates the power of structured reasoning over traditional RL:

  • Zero-Shot Superiority: SPRING (using GPT-4) achieved a game score of 27.3%, nearly doubling the 14.5% achieved by DreamerV3, a state-of-the-art RL model trained for 1 million steps.
  • Tech Tree Mastery: SPRING reached a maximum achievement depth of 6, whereas RL baselines often plateaued earlier due to the difficulty of discovering complex sequences (like crafting a furnace to smelt iron).
  • Consistency: The QA-DAG structure significantly improved reliability, reducing performance variance from over 60% (with simple prompting) to just 6%.

Conclusion

SPRING demonstrates that LLMs, when provided with structured context and a rigorous reasoning framework, can outperform specialized RL agents in complex environments without any training. While challenges remain—such as the high inference cost of GPT-4 and the current reliance on text-based visual descriptors—the success of SPRING suggests a promising future for hybrid AI systems that combine high-level LLM reasoning with low-level control.

Summary updated

Abstract

Open-world survival games pose significant challenges for AI algorithms due to their multi-tasking, deep exploration, and goal prioritization requirements. Despite reinforcement learning (RL) being popular for solving games, its high sample complexity limits its effectiveness in complex open-world games like Crafter or Minecraft. We propose a novel approach, SPRING, to read the game's original academic paper and use the knowledge learned to reason and play the game through a large language model (LLM). Prompted with the LaTeX source as game context and a description of the agent's current observation, our SPRING framework employs a directed acyclic graph (DAG) with game-related questions as nodes and dependencies as edges. We identify the optimal action to take in the environment by traversing the DAG and calculating LLM responses for each node in topological order, with the LLM's answer to final node directly translating to environment actions. In our experiments, we study the quality of in-context "reasoning" induced by different forms of prompts under the setting of the Crafter open-world environment. Our experiments suggest that LLMs, when prompted with consistent chain-of-thought, have great potential in completing sophisticated high-level trajectories. Quantitatively, SPRING with GPT-4 outperforms all state-of-the-art RL baselines, trained for 1M steps, without any training. Finally, we show the potential of games as a test bed for LLMs.
BibTeX
@misc{wu2023spring,
  author = {Yue Wu and So-Yeon Min and Shrimai Prabhumoye and Yonatan Bisk and Ruslan Salakhutdinov and Amos Azaria and Tom M. Mitchell and Yuanzhi Li},
  title = {{SPRING: Studying the Paper and Reasoning to Play Games}},
  year = {2023},
  eprint = {2305.15486},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.15486},
}
Figure