Introduction
Reinforcement Learning (RL) has demonstrated remarkable success in mastering complex environments, ranging from board games like Go to various Atari 2600 titles. However, these successes are often overshadowed by extreme sample complexity. For instance, state-of-the-art agents for the game Skiing have historically required upwards of 80 billion frames—equivalent to roughly 100 years of continuous play—to reach mastery.
This inefficiency stems from a fundamental difference between artificial agents and humans: the lack of prior knowledge. While a human player can read a manual to understand that a tree is an obstacle and a gate is a goal, a standard RL agent must collide with the tree thousands of times to learn its negative utility. This paper introduces the Read and Reward (RnR) framework, which leverages Large Language Models (LLMs) to bridge this gap by extracting actionable knowledge from game manuals and converting it into auxiliary reward signals.
The Read and Reward (RnR) Framework
The RnR framework automates the process of turning unstructured text into reinforcement signals through a three-stage pipeline:
1. QA Extraction (The Reader)
Instruction manuals are often lengthy and contain irrelevant information. To handle this, the system employs a RoBERTa model fine-tuned on the SQuAD dataset. The "Reader" identifies objects within the game (using TF-IDF) and asks targeted questions such as "What is the objective?" or "What happens when the player hits [object]?" to extract relevant text snippets.
2. Reasoning and Reward Mapping (The Thinker)
The extracted snippets are processed by Macaw, a zero-shot logic-capable LLM. The system prompts the model with reasoning tasks, such as: "Should you hit [object] if you want to win?" The LLM’s binary response is then mapped to a numerical auxiliary reward (typically +5 for positive interactions and -5 for negative ones). This allows the agent to benefit from human common sense without manual reward engineering.
3. Interaction Detection and Grounding
During gameplay, the agent must determine when it has interacted with an object to trigger the auxiliary reward. The authors tested two approaches:
- Ground-Truth Grounding: Using object locations directly from game memory.
- Full Pipeline Grounding: Utilizing SPACE (unsupervised object detection) and CLIP (zero-shot image-to-text grounding) to identify objects and detect interactions purely from pixels.
Experimental Evaluation
The framework was evaluated on four Atari games: Skiing, Tennis, Ms. Pac-Man, and Breakout. Textual knowledge was sourced from official manuals on atariage.com and community-contributed "Gameplay" sections on Wikipedia.
Key Results
The integration of textual knowledge led to dramatic improvements in learning efficiency:
- Sample Efficiency: In Breakout, an Agent57 model equipped with RnR reached baseline performance levels 112x faster.
- Sparse Reward Mastery: In Skiing, where rewards are notoriously delayed, standard A2C and PPO baselines often fail to learn. The RnR agent achieved a 60% performance boost while using 1000x fewer frames than previous record-holders.
- Source Robustness: The framework performed consistently regardless of whether it used professional manuals or Wikipedia entries, suggesting that the core logic required for gameplay is robustly represented across different writing styles.
Discussion and Limitations
While RnR represents a significant step forward, several challenges remain. The "full pipeline" relying on vision models like CLIP is currently brittle; misidentifying a game object can lead to incorrect reward signals that hinder learning. Furthermore, the current implementation focuses on simple "hit" interactions. It does not yet capture complex conditional instructions, such as "only collect the key after the door is unlocked." Finally, the distance-based interaction detection is optimized for 2D environments and would require more sophisticated spatial reasoning for 3D applications.
Conclusion
The Read and Reward framework demonstrates that LLMs can effectively serve as reward designers, translating human language into a format that RL agents can exploit. By jumpstarting the learning process with prior knowledge, RnR moves RL closer to the human-like ability to learn new tasks efficiently through instruction.
Summary updated