TL;DR
This paper introduces a novel approach for AI agents to play interactive fiction games, specifically focusing on the TextWorld Commonsense (TWC) benchmark. By leveraging GPT-4 with a carefully designed prompt that incorporates Chain-of-Thought, few-shot examples, and feedback augmentation, the authors achieve state-of-the-art performance without any fine-tuning, demonstrating the power of LLMs for complex reasoning tasks.
What Problem Does This Paper Address?
Interactive fiction games, like those found in TextWorld, present a significant challenge for AI agents. These games require not only understanding natural language commands and game engine responses but also possessing common sense knowledge to make appropriate decisions, such as knowing where to place an item. Traditional AI approaches often rely on reinforcement learning, which can be difficult and time-consuming to train, especially given the sparse rewards inherent in these games. The authors aim to bridge this gap by exploring how large language models (LLMs), particularly GPT-4, can efficiently tackle these games, bypassing the need for extensive training and leveraging their inherent world knowledge.
Methods
The core of this research lies in a prompt engineering strategy designed to maximize GPT-4's capabilities for interactive fiction. The approach combines several key techniques:
- Chain-of-Thought (CoT): By prompting GPT-4 to "think step by step," the model is encouraged to generate a reasoning process before outputting its final action, leading to more coherent and accurate decisions.
- Few-Shot Prompting (specifically, one-shot): The prompt includes an example walkthrough of a game scenario. This provides the LLM with a concrete illustration of how to play, helping it understand the task and expected behavior.
- Output Template: A structured output format is used, guiding GPT-4 to provide specific sections like "Question," "Consideration," and "Next Action." This standardization is crucial for automatically extracting the agent's chosen action.
- Feedback Augmentation: This is a novel contribution. The authors noticed that the TWC game engine's feedback for incorrect item placements was ambiguous. They augment this feedback by explicitly adding phrases like "Right position" for correct placements and "Wrong position, you should put it somewhere else, maybe the other room" for incorrect ones. This provides GPT-4 with clearer signals about the success or failure of its actions.
The authors utilize the OpenAI API to interact with GPT-4 Turbo and GPT-3.5 Turbo, without any fine-tuning.
Datasets
The primary dataset used for evaluation is TextWorld Commonsense (TWC). This dataset consists of games designed to test an agent's ability to apply common sense knowledge. It includes five games each for easy, medium, and hard difficulty levels, varying in the number of items and rooms. The authors note that TWC is built upon the TextWorld environment. No new datasets were introduced in this paper.
Key Insights and Ideas
- LLMs as powerful game players: The results strongly suggest that LLMs like GPT-4, when properly prompted, can achieve impressive performance in complex reasoning tasks like interactive fiction games, rivaling or surpassing traditional RL agents.
- Prompt engineering is key: The success of the approach hinges on the carefully crafted prompt. The combination of CoT, one-shot examples, and output templates significantly structures the LLM's output and reasoning.
- Feedback augmentation is crucial for ambiguous environments: The TWC environment's feedback can be subtle. Explicitly augmenting this feedback, especially for item placement, provides a significant performance boost, demonstrating that even LLMs benefit from clearer signals.
- GPT-4's superiority over GPT-3.5: The ablation study highlights GPT-4's significant advantage, particularly in hard-level games, where GPT-3.5 struggles to produce valid outputs and achieves much lower scores.
Key Results
The proposed GPT-4 approach with feedback augmentation achieved state-of-the-art results on the TWC dataset:
- Easy and Medium Levels: GPT-4 with the full prompt successfully completed all tasks.
- Hard Levels: The approach achieved a normalized score of 0.70, significantly outperforming the best baseline score of 0.57 (achieved by the TWC Agent, a reinforcement learning model).
- Ablation Study:
- Replacing GPT-4 with GPT-3.5 resulted in a drastic drop in performance on hard-level games to 0.18.
- Removing feedback augmentation ("-FA") reduced the hard-level score from 0.70 to 0.52.
- Removing Chain-of-Thought ("-COT") and one-shot examples ("-one-shot") also led to performance drops, though less severe than removing GPT-4 or feedback augmentation.
Limitations and Future Work
- GPT-3.5 limitations: The paper notes that GPT-3.5 Turbo often produces unstructured outputs that require manual parsing, making it less efficient than GPT-4 for this task.
- Sample selection impact: The performance of the one-shot prompting technique is sensitive to the quality and relevance of the example walkthrough provided.
- Model's reasoning in complex scenarios: While GPT-4 performs well, it can still make mistakes, especially in hard-level games where it might struggle with inferring the correct location for an item if the refrigerator, for instance, is in another room. The paper suggests that providing hints like "no need to close anything" could further improve performance but might compromise generality.
- Future Work: The authors suggest exploring more complex interactive fiction environments like Complex World and investigating how to further improve the LLM's reasoning capabilities in scenarios with more intricate spatial relationships and item dependencies. They also hint at the potential for incorporating dataset-specific knowledge to further boost performance.
Summary updated