Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manuals

Yue Wu, Yewen Fan, Paul Pu Liang, Amos Azaria, Yuanzhi Li, Tom Mitchell

TL;DR

This paper introduces "Read and Reward," a framework that allows Reinforcement Learning (RL) agents to learn Atari games more efficiently by reading and understanding instruction manuals. By extracting relevant information and using it to provide auxiliary rewards, their A2C agent achieves significant performance gains and drastically reduces training time, even surpassing state-of-the-art methods on challenging games.

What Problem Does This Paper Address?

Reinforcement Learning (RL) agents, despite impressive successes, face a major hurdle: sample complexity. This means they often require an enormous amount of interaction with an environment to learn effectively. For instance, solving the Atari game "Skiing" can take billions of frames, equivalent to years of real-time play. Humans, on the other hand, learn much faster by leveraging readily available unstructured text like instruction manuals. This paper addresses the gap between human and RL learning efficiency by exploring how RL agents can benefit from reading and understanding these human-readable documents to accelerate their learning process.

Methods

The core of the proposed solution is the "Read and Reward" framework, which has two main components:

  • QA Extraction Module: This module takes the raw instruction manual text and a set of pre-defined questions (e.g., "What is the objective of the game?", "What happens when the player hits a ghost?"). It uses an extractive Question Answering (QA) model (specifically, a fine-tuned RoBERTa) to identify and extract relevant text snippets that answer these questions. This effectively summarizes the manual, tackling the challenge of its sheer length and redundancy.
  • Reasoning Module: This module leverages the extracted information to provide auxiliary rewards to the RL agent. It uses a large language model (Macaw) to reason about object-agent interactions. For example, if the QA module identifies that hitting a "tree" in "Skiing" is bad, the reasoning module can infer that an interaction between the agent and a tree should result in a negative auxiliary reward. This auxiliary reward is then fed to a standard A2C RL agent.

The framework also incorporates object detection and grounding to identify in-game objects and their interactions, which are then used to query the Reasoning Module.

Datasets

The experiments were conducted on several Atari games, including:

  • Skiing
  • Tennis
  • Ms. Pac-Man
  • Breakout

The primary data source for the framework is the instruction manuals for these Atari games, obtained from sources like atariage.com and Wikipedia. The paper also mentions using ground-truth object locations for some games to bypass the challenges of unsupervised object detection.

Key Insights and Ideas

  • Manuals as a Rich Source of Knowledge: The paper strongly supports the hypothesis that human-written instruction manuals contain valuable information that can significantly boost RL agent performance and efficiency, especially in games with sparse rewards.
  • QA as an Effective Summarization Tool: Using extractive QA to process lengthy manuals is an effective way to distill relevant information, overcoming the input length limitations of many language models.
  • LLMs for In-Context Reasoning: Leveraging the in-context reasoning capabilities of large language models (LLMs) allows the framework to infer the implications of in-game events (like hitting an obstacle) without explicit task-specific training.
  • Auxiliary Rewards Drive Efficiency: The auxiliary rewards generated by the Read and Reward framework are crucial for accelerating learning. The paper shows a strong positive correlation between these auxiliary rewards and the game score, indicating that optimizing for them directly benefits the agent's performance.
  • Generalizability Across Manual Sources: The framework demonstrates consistent performance whether using official Atari manuals or Wikipedia entries, highlighting its robustness to variations in text style and detail.

Key Results

  • Significant Speedup: In the "Skiing" game, the Read and Reward-assisted A2C agent achieved performance comparable to the SOTA Agent 57 but required over 1000x fewer training frames.
  • Improved Performance on Sparse Rewards: The framework improved performance on 4 Atari games with sparse rewards.
  • Efficiency Gains: Figure 5 shows that the Read and Reward agent on "Skiing" is not only efficient in terms of frames seen but also achieves a competitive game score compared to baselines like A2C and MuZero.
  • Correlation with Game Score: Figure 6 demonstrates that the auxiliary reward generated by the framework has a strong positive correlation with the actual game score, suggesting the agent effectively utilizes the provided guidance.

Limitations and Future Work

  • Object Localization and Detection: A significant implementation challenge is reliably detecting and grounding objects in the Atari environment, especially for games with complex visuals or overlapping objects. The paper notes that current unsupervised techniques lack reliability for this.
  • Interaction Simplification: The framework currently simplifies interactions by focusing on proximity and ignoring interactions that don't involve the agent. This might not scale to more complex 3D environments.
  • Future Directions: The authors suggest future work could focus on improving object detection and grounding, expanding the types of interactions considered, and exploring environments where object localization is more readily available or where more sophisticated object detection models can be used. They also envision future work benefiting from advancements in video-language models for more robust grounding.

Summary updated

Abstract

High sample complexity has long been a challenge for RL. On the other hand, human learn to perform tasks not only from interaction or demonstrations, but also by reading unstructured text documents, e.g., instruction manuals. Instruction manuals and wiki pages are among the most abundant data that could inform agents of valuable features and policies or task-specific environmental dynamics and reward structures. Therefore, we hypothesize that the ability to utilize human-written instruction manuals to assist learning policies for specific tasks should lead to a more efficient and better-performing agent. We propose the Read and Reward framework. Read and Reward speeds up RL algorithms on Atari games by reading manuals released by the Atari game developers. Our framework consists of a QA Extraction module that extracts and summarizes relevant information from the manual and a Reasoning module that evaluates object-agent interactions based on information from the manual. Auxiliary reward is then provided to a standard A2C RL agent, when interaction is detected. When assisted by our design, A2C improves on 4 games in the Atari environment with sparse rewards, and requires 1000x less training frames compared to the previous SOTA Agent 57 on Skiing, the hardest game in Atari.
BibTeX
@inproceedings{wu2023read,
  author = {Yue Wu and Yewen Fan and Paul Pu Liang and Amos Azaria and Yuanzhi Li and Tom Mitchell},
  title = {{Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manuals}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=qP0Drg2HuH},
}
Figure