SPRING: Studying the Paper and Reasoning to Play Games

Yue Wu, Shrimai Prabhumoye, So Yeon Min, Yonatan Bisk, Ruslan Salakhutdinov, Amos Azaria, Tom Mitchell, Yuanzhi Li

Open-world survival games like Minecraft and its research-oriented cousin Crafter represent a "final boss" for modern AI. To succeed, an agent must juggle conflicting goals, navigate complex technology trees, and explore deep environments where rewards are few and far between. Traditionally, we’ve thrown Reinforcement Learning (RL) at these problems, but RL is notoriously "data-hungry," often requiring millions of steps of trial-and-error just to learn how to craft a basic tool.

A new paper from researchers at Carnegie Mellon, NVIDIA, Ariel University, and Microsoft Research proposes a different path. Instead of letting an agent wander aimlessly until it bumps into a reward, why not just let it read the manual? Their framework, SPRING (Studying the Paper and Reasoning to Play Games), allows a Large Language Model (LLM) to study the original academic paper describing a game and use that knowledge to outplay state-of-the-art RL agents—zero-shot.

What Problem Does This Paper Address?

The core challenge is sample complexity. RL agents start as "blank slates." In a game like Crafter, an agent might have to perform a specific sequence of five or six actions (find wood, craft table, craft pickaxe, find stone, etc.) before receiving a single point of reward. Finding this sequence through random exploration is like finding a needle in a haystack.

Humans don't play games this way; we read guides or manuals. While LLMs like GPT-4 have immense reasoning capabilities, they struggle with long-horizon planning in dynamic environments. They can get "distracted" by irrelevant details or lose the thread of their own reasoning over hundreds of game steps. SPRING aims to bridge this gap by structuring how an LLM extracts knowledge from a paper and how it reasons through each move.

Methods

The SPRING framework operates in two distinct stages:

1. Studying the Paper

Rather than dumping an entire academic paper into a prompt, SPRING uses a three-step extraction pipeline. It parses the LaTeX source of the environment's original paper and uses an LLM to filter paragraphs for relevance. It then extracts key information about game mechanics, objectives, and item dependencies (the "tech tree") into a condensed context string.

Figure 2: Paper Studying Moudle. The 3-step approach for computing Cq from the LATEXsource code of Hafner (2021). First, as shown in the left column, for each paragraph we compute LLM answer for all relevancy questions in Qrel, and keep only the relevant paragraphs. Second, as shown in the middle column, we compute paragraph-level LLM answer to q. Third, we summarize the answer into Cq with a summary prompt; we concatenate Cq across q ∈Qgame and obtain C.

2. Reasoning via QA-DAG

This is the framework's most significant innovation. To prevent the LLM from losing focus, SPRING employs a Directed Acyclic Graph (DAG) of questions. At every environment step, the LLM must answer a series of nine questions in a specific order.

For example, question $q_7$ ("List top 5 actions the player should take") depends on the answer to $q_5$ ("List top 3 sub-tasks"). By traversing this graph, the model's "Chain-of-Thought" is explicitly regulated. The final node in the DAG translates directly into a game action. This structure significantly reduces reasoning errors and ensures the agent's actions are consistent with its high-level goals.

Figure 1: Overview of SPRING. The context string, shown in the middle column, is obtained by parsing the LATEX source code of Hafner (2021). The LLM-based agent then takes input from a visual game descriptor and the context string. The agent uses questions composed into a DAG for chain-of-thought reasoning, and the last node of the DAG is parsed into action.

Datasets

The researchers evaluated SPRING on Crafter, an open-world survival benchmark. Crafter is designed to mimic the challenges of Minecraft but in a 2D grid-world format that allows for faster iteration. The "knowledge base" for the agent was the LaTeX source of the original Crafter paper (Hafner, 2021).

Key Insights and Ideas

  • Reading is Faster than Wandering: The most striking insight is that an LLM can achieve superior performance with zero training steps simply by leveraging human-written documentation.
  • Structured Reasoning is Mandatory: The authors found that standard "Chain-of-Thought" prompting (just asking the model to "think step-by-step") was too inconsistent for long games. The QA-DAG structure was essential to keep the model's logic grounded in the current state of the game.
  • LLMs are Robust to Bad Info: In ablation studies, when researchers manually removed key information from the context (like the requirement for a crafting table), GPT-4 was often able to "recover" by guessing and trying different interactions until it succeeded.

Key Results

SPRING, powered by GPT-4, didn't just compete with RL—it dominated.

  • Zero-Shot vs. 1 Million Steps: SPRING achieved a game score of 27.3%, nearly doubling the 14.5% achieved by DreamerV3, the previous state-of-the-art RL baseline, which required 1 million steps of training.
  • Tech Tree Mastery: SPRING was over 10x more successful than RL baselines at reaching deep tech-tree milestones, such as "Collect Iron" or "Make Stone Pickaxe," because it understood the prerequisites from the start.
  • Model Sensitivity: The choice of LLM matters. SPRING with GPT-4 performed exceptionally well, while GPT-3.5 struggled to follow the fine-grained instructions in the DAG, resulting in a 73% performance drop.

Limitations and Future Work

Despite its success, SPRING has clear hurdles to overcome:

  • The Vision Gap: SPRING currently relies on a "Visual Descriptor"—a separate piece of code that uses computer vision filters to turn the game screen into a text description. The authors suggest that future iterations could use multi-modal Vision-Language Models (VLMs) to perceive the game directly.
  • Inference Cost: Reasoning through a DAG of nine questions at every step is expensive. The researchers estimated a cost of roughly $270 per game using current GPT-4 API pricing.
  • Static Knowledge: SPRING doesn't "learn" from its mistakes in the way RL does; it only reasons based on its fixed context. A hybrid approach that combines the high-level reasoning of SPRING with the low-level learning of RL could be the next frontier in open-world AI.

Summary updated

Abstract

Open-world survival games pose significant challenges for AI algorithms due to their multi-tasking, deep exploration, and goal prioritization requirements. Despite reinforcement learning (RL) being popular for solving games, its high sample complexity limits its effectiveness in complex open-world games like Crafter or Minecraft. We propose a novel approach, SPRING, to read the game's original academic paper and use the knowledge learned to reason and play the game through a large language model (LLM). Prompted with the LaTeX source as game context and a description of the agent's current observation, our SPRING framework employs a directed acyclic graph (DAG) with game-related questions as nodes and dependencies as edges. We identify the optimal action to take in the environment by traversing the DAG and calculating LLM responses for each node in topological order, with the LLM's answer to final node directly translating to environment actions. In our experiments, we study the quality of in-context "reasoning" induced by different forms of prompts under the setting of the Crafter open-world environment. Our experiments suggest that LLMs, when prompted with consistent chain-of-thought, have great potential in completing sophisticated high-level trajectories. Quantitatively, SPRING with GPT-4 outperforms all state-of-the-art RL baselines, trained for 1M steps, without any training. Finally, we show the potential of games as a test bed for LLMs.
BibTeX
@misc{wu2023spring,
  author = {Yue Wu and Shrimai Prabhumoye and So Yeon Min and Yonatan Bisk and Ruslan Salakhutdinov and Amos Azaria and Tom Mitchell and Yuanzhi Li},
  title = {{SPRING: Studying the Paper and Reasoning to Play Games}},
  year = {2023},
  eprint = {2305.15486},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.15486},
}
Figure