Abstract
Open-world survival games pose significant challenges for AI algorithms due to their multi-tasking, deep exploration, and goal prioritization requirements. Despite reinforcement learning (RL) being popular for solving games, its high sample complexity limits its effectiveness in complex open-world games like Crafter or Minecraft. We propose a novel approach, SPRING, to read the game's original academic paper and use the knowledge learned to reason and play the game through a large language model (LLM). Prompted with the LaTeX source as game context and a description of the agent's current observation, our SPRING framework employs a directed acyclic graph (DAG) with game-related questions as nodes and dependencies as edges. We identify the optimal action to take in the environment by traversing the DAG and calculating LLM responses for each node in topological order, with the LLM's answer to final node directly translating to environment actions. In our experiments, we study the quality of in-context "reasoning" induced by different forms of prompts under the setting of the Crafter open-world environment. Our experiments suggest that LLMs, when prompted with consistent chain-of-thought, have great potential in completing sophisticated high-level trajectories. Quantitatively, SPRING with GPT-4 outperforms all state-of-the-art RL baselines, trained for 1M steps, without any training. Finally, we show the potential of games as a test bed for LLMs.
Introduction
Reinforcement Learning (RL) has long been the gold standard for mastering games, yet it faces significant hurdles: high sample complexity and difficulty incorporating prior knowledge. RL agents typically start as a "blank slate," requiring millions of trials to discover basic mechanics that a human could learn in minutes by reading a manual. In complex, open-world survival games like Crafter—a 2D Minecraft-inspired benchmark—agents must navigate deep technology trees and prioritize conflicting goals. Traditional RL agents often struggle to reach late-game achievements because the probability of stumbling upon the correct sequence of actions through random exploration is infinitesimally small.
The SPRING framework addresses this by asking: Can we skip millions of training steps by simply letting an AI read the "instruction manual"—the academic paper that introduced the game? The primary benchmark used is Crafter, and the knowledge source is the LaTeX source of the paper: “Benchmarking the Spectrum of Agent Capabilities” (Hafner, 2021). Notably, Crafter was released after the training data cutoff for GPT-3.5 and GPT-4, ensuring no prior knowledge of the game mechanics.
Methods
The SPRING framework consists of a two-stage process: studying the paper and reasoning through a Question-Answer Directed Acyclic Graph (QA-DAG).
Studying the Paper
The system parses the LaTeX source code of the original Crafter environment paper. It uses an LLM to filter out irrelevant sections and extract key gameplay information, such as object interactions and tech-tree dependencies. This process involves filtering text paragraph-by-paragraph and extracting relevant information through targeted QA.
Reasoning via QA-DAG
To play the game, SPRING uses a QA-DAG. At each time step, the LLM is asked a series of 9 structured questions in topological order. These range from "What do I see?" and "Did my last action succeed?" to "What are my top 3 sub-tasks?" and finally, "Choose the best executable action."
A Visual Descriptor translates the game's pixel grid into a text-based description, allowing the LLM to perceive the world. By forcing the LLM to reason through the DAG, the framework ensures a consistent "chain-of-thought" across hundreds of gameplay steps, reducing performance variance.
Results and Insights
SPRING demonstrates that for tasks where rules can be described in text, reasoning over a manual is more effective than a million steps of trial-and-error.
- Performance: SPRING with GPT-4 achieved a game score of 27.3%, nearly doubling the 14.5% achieved by DreamerV3, the previous state-of-the-art RL algorithm.
- Zero-Shot Capability: While RL baselines required 1 million training steps, SPRING required zero training steps.
- Tech Tree Depth: SPRING reached depth 6 achievements (e.g., "Collect Iron"), whereas most RL baselines struggle past depth 4.
- Model Comparison: GPT-4 was essential; switching to GPT-3.5 caused the score to plummet to 3.3%, highlighting the necessity of advanced reasoning capabilities.
Discussion and Future Work
While SPRING outperforms RL baselines, it faces several limitations. It currently relies on a hand-coded visual descriptor; future iterations could utilize Vision-Language Models (VLMs) for direct pixel perception. Additionally, the cost of using the GPT-4 API is high, approximately $270 per game. Finally, SPRING's knowledge is static; it does not "learn" from interaction like RL. A hybrid approach combining LLM reasoning with RL's interactive learning presents a promising future direction.
Summary updated