Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manuals

Yue Wu, Yewen Fan, Paul Pu Liang, Amos Azaria, Yuanzhi Li, Tom M. Mitchell

TL;DR

The Read and Reward (R&R) framework enables Reinforcement Learning (RL) agents to significantly improve their sample efficiency by "reading" human-written instruction manuals and Wikipedia pages. By using Large Language Models (LLMs) to extract and reason over game objectives and object interactions, the system provides auxiliary rewards that help agents master complex Atari games like Skiing and Breakout up to 112x faster than standard baselines.

What Problem Does This Paper Address?

Reinforcement Learning has achieved remarkable feats, but it remains notoriously "data-hungry." For instance, the state-of-the-art Agent57 required 80 billion frames—roughly 100 years of non-stop play—to solve the Atari game Skiing. Humans, by contrast, can master such games in minutes by leveraging prior knowledge or reading a simple instruction manual.

The core challenge is sample complexity: agents typically start with zero knowledge and must discover every reward through grueling trial and error. While previous research has tried using synthetic, template-based language to guide agents, these methods don't generalize to the messy, unstructured text found in real-world manuals or wiki pages. This paper tackles the gap between raw human text and RL reward signals.

Methods

The authors propose the Read and Reward framework, which bridges the gap between unstructured text and RL agents through a two-step pipeline:

  1. QA Extraction Module (Read): Human manuals are often long and filled with irrelevant legal or technical jargon. This module uses a RoBERTa-large model (fine-tuned on SQuAD) to extract relevant snippets. It asks generic questions (e.g., "What is the objective?") and object-specific questions (e.g., "What happens when the player hits a tree?") to create a concise summary.
  2. Reasoning Module (Reward): A zero-shot LLM (Macaw) takes the extracted context and reasons about specific in-game events. For every detected interaction, it answers a binary question: "Should you hit [object] if you want to win?" The "Yes/No" answers are mapped to $+5$ or $-5$ auxiliary rewards.
Figure 1: An overview of our Read and Reward framework. Our system receives the current frame in the environment, and the instruction manual as input. After object detection and grounding, the QA Extraction Module extracts and summarizes relevant information from the manual, and the Reasoning Module assigns auxiliary rewards to detected in-game events by reasoning with outputs from the QA Extraction Module. The “Yes/No” answers are then mapped to +5/ −5 auxiliary rewards.

To connect these text-based rewards to the visual game environment, the authors use SPACE for unsupervised object detection and CLIP for zero-shot grounding (associating visual clusters with text keywords like "tree" or "skier"). When the agent's bounding box intersects with an object, the reasoning module's pre-calculated reward is triggered.

Figure 3: Illustration of the Reasoning Module. The <context> (from Figure 2) related to the ob- ject ghost from the QA Extraction module is con- catenated with a template-generated question to form the zero-shot in-context reasoning prompt for a Large Language Model. The Yes/No answer from the LLM is then turned into an auxiliary re- ward for the agent.

Datasets

The framework was evaluated on four Atari games: Skiing, Tennis, Ms. Pac-Man, and Breakout. The researchers used two distinct sources for text data:

  • Official Atari Manuals: Scanned and parsed HTML versions of the original developer manuals.
  • Wikipedia: The "Gameplay" sections for each game, which are typically much shorter and more concise than official manuals.

Key Insights and Ideas

  • Extractive QA as a Filter: LLMs have limited context windows and can get "distracted" by long documents. Using a specialized extractive QA model to pre-filter the manual into a "dialogue" format makes the reasoning task much easier for the subsequent LLM.
  • Zero-Shot Reasoning for Implicit Rewards: Manuals rarely say "hitting a tree gives -5 points." Instead, they say "hitting a tree loses time" and "the goal is the fastest time." The LLM successfully connects these dots to realize that hitting a tree is a negative event.
  • Robustness Across Sources: The system performed consistently whether using the detailed official manuals or the brief Wikipedia summaries, suggesting that LLMs are capable of extracting the "essence" of gameplay from various writing styles.

Key Results

The impact of adding human-readable knowledge is dramatic, especially in "delayed reward" scenarios where the environment only gives a score at the very end of the game.

  • Efficiency in Skiing: The R&R agent achieved a 60% performance improvement over a random baseline using 1000x fewer frames than the SOTA Agent57.
  • Massive Speedups: In Breakout, the R&R-assisted Agent57 reached the same performance as the baseline 112x faster.
  • Performance Boost: Across all tested games and various RL algorithms (A2C, PPO, R2D1, Agent57), the version with "Read and Reward" consistently outperformed the baseline.

Limitations and Future Work

While the results are impressive, the authors highlight a major bottleneck: Grounding. The SPACE/CLIP pipeline for detecting and naming objects in the game is not yet 100% reliable. For example, if two objects overlap, the system might misidentify them, leading to incorrect rewards.

Figure 4: Examples of SPACE (Lin et al., 2020) and CLIP (Radford et al., 2021) in the full end- to-end pipeline (Section 4.3). The top row shows bounding boxes for objects and the bottom row shows corresponding object masks as detected by SPACE. Most of the bounding boxes generated are correct. Left: SPACE confuses bounding boxes of agent and tree into one and the box gets classified as “tree” (blue), and the auxiliary penalty is not properly triggered. Right: The flag next to the agent (in red circle) is not detected, and therefore the auxiliary reward is not provided.

Additionally, the current system only considers "hit" interactions based on 2D distance. Future work could involve more complex interaction types (like "collecting" vs "avoiding") and extending the framework to 3D environments like Minecraft, where spatial reasoning becomes even more critical.

Summary updated

Abstract

High sample complexity has long been a challenge for RL. On the other hand, humans learn to perform tasks not only from interaction or demonstrations, but also by reading unstructured text documents, e.g., instruction manuals. Instruction manuals and wiki pages are among the most abundant data that could inform agents of valuable features and policies or task-specific environmental dynamics and reward structures. Therefore, we hypothesize that the ability to utilize human-written instruction manuals to assist learning policies for specific tasks should lead to a more efficient and better-performing agent. We propose the Read and Reward framework. Read and Reward speeds up RL algorithms on Atari games by reading manuals released by the Atari game developers. Our framework consists of a QA Extraction module that extracts and summarizes relevant information from the manual and a Reasoning module that evaluates object-agent interactions based on information from the manual. An auxiliary reward is then provided to a standard A2C RL agent, when interaction is detected. Experimentally, various RL algorithms obtain significant improvement in performance and training speed when assisted by our design.
BibTeX
@inproceedings{wu2023read,
  author = {Yue Wu and Yewen Fan and Paul Pu Liang and Amos Azaria and Yuanzhi Li and Tom M. Mitchell},
  title = {{Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manuals}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=qP0Drg2HuH},
}
Figure