Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manuals

Yue Wu, Yewen Fan, Paul Pu Liang, Amos Azaria, Yuanzhi Li, Tom M. Mitchell

Introduction

Reinforcement Learning (RL) has demonstrated remarkable success in mastering complex environments, ranging from board games like Go to various Atari 2600 titles. However, these successes are often overshadowed by extreme sample complexity. For instance, state-of-the-art agents for the game Skiing have historically required upwards of 80 billion frames—equivalent to roughly 100 years of continuous play—to reach mastery.

This inefficiency stems from a fundamental difference between artificial agents and humans: the lack of prior knowledge. While a human player can read a manual to understand that a tree is an obstacle and a gate is a goal, a standard RL agent must collide with the tree thousands of times to learn its negative utility. This paper introduces the Read and Reward (RnR) framework, which leverages Large Language Models (LLMs) to bridge this gap by extracting actionable knowledge from game manuals and converting it into auxiliary reward signals.

The Read and Reward (RnR) Framework

The RnR framework automates the process of turning unstructured text into reinforcement signals through a three-stage pipeline:

1. QA Extraction (The Reader)

Instruction manuals are often lengthy and contain irrelevant information. To handle this, the system employs a RoBERTa model fine-tuned on the SQuAD dataset. The "Reader" identifies objects within the game (using TF-IDF) and asks targeted questions such as "What is the objective?" or "What happens when the player hits [object]?" to extract relevant text snippets.

2. Reasoning and Reward Mapping (The Thinker)

The extracted snippets are processed by Macaw, a zero-shot logic-capable LLM. The system prompts the model with reasoning tasks, such as: "Should you hit [object] if you want to win?" The LLM’s binary response is then mapped to a numerical auxiliary reward (typically +5 for positive interactions and -5 for negative ones). This allows the agent to benefit from human common sense without manual reward engineering.

3. Interaction Detection and Grounding

During gameplay, the agent must determine when it has interacted with an object to trigger the auxiliary reward. The authors tested two approaches:

  • Ground-Truth Grounding: Using object locations directly from game memory.
  • Full Pipeline Grounding: Utilizing SPACE (unsupervised object detection) and CLIP (zero-shot image-to-text grounding) to identify objects and detect interactions purely from pixels.

Experimental Evaluation

The framework was evaluated on four Atari games: Skiing, Tennis, Ms. Pac-Man, and Breakout. Textual knowledge was sourced from official manuals on atariage.com and community-contributed "Gameplay" sections on Wikipedia.

Key Results

The integration of textual knowledge led to dramatic improvements in learning efficiency:

  • Sample Efficiency: In Breakout, an Agent57 model equipped with RnR reached baseline performance levels 112x faster.
  • Sparse Reward Mastery: In Skiing, where rewards are notoriously delayed, standard A2C and PPO baselines often fail to learn. The RnR agent achieved a 60% performance boost while using 1000x fewer frames than previous record-holders.
  • Source Robustness: The framework performed consistently regardless of whether it used professional manuals or Wikipedia entries, suggesting that the core logic required for gameplay is robustly represented across different writing styles.

Discussion and Limitations

While RnR represents a significant step forward, several challenges remain. The "full pipeline" relying on vision models like CLIP is currently brittle; misidentifying a game object can lead to incorrect reward signals that hinder learning. Furthermore, the current implementation focuses on simple "hit" interactions. It does not yet capture complex conditional instructions, such as "only collect the key after the door is unlocked." Finally, the distance-based interaction detection is optimized for 2D environments and would require more sophisticated spatial reasoning for 3D applications.

Conclusion

The Read and Reward framework demonstrates that LLMs can effectively serve as reward designers, translating human language into a format that RL agents can exploit. By jumpstarting the learning process with prior knowledge, RnR moves RL closer to the human-like ability to learn new tasks efficiently through instruction.

Summary updated

Abstract

High sample complexity has long been a challenge for RL. On the other hand, humans learn to perform tasks not only from interaction or demonstrations, but also by reading unstructured text documents, e.g., instruction manuals. Instruction manuals and wiki pages are among the most abundant data that could inform agents of valuable features and policies or task-specific environmental dynamics and reward structures. Therefore, we hypothesize that the ability to utilize human-written instruction manuals to assist learning policies for specific tasks should lead to a more efficient and better-performing agent. We propose the Read and Reward framework. Read and Reward speeds up RL algorithms on Atari games by reading manuals released by the Atari game developers. Our framework consists of a QA Extraction module that extracts and summarizes relevant information from the manual and a Reasoning module that evaluates object-agent interactions based on information from the manual. Auxiliary reward is then provided to a standard A2C RL agent, when interaction is detected. When assisted by our design, A2C improves on 4 games in the Atari environment with sparse rewards, and requires 1000x less training frames compared to the previous SOTA Agent 57 on Skiing, the hardest game in Atari.
BibTeX
@inproceedings{wu2023read,
  author = {Yue Wu and Yewen Fan and Paul Pu Liang and Amos Azaria and Yuanzhi Li and Tom M. Mitchell},
  title = {{Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manuals}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=qP0Drg2HuH},
}
Figure