Introduction
Standard Reinforcement Learning (RL) agents are notoriously brittle. Train an agent to play a game, and it will likely master it. However, slightly change the rules—such as making a "poison" monster weak to "fire" instead of "ice"—and the agent's performance usually collapses. To address this, researchers have developed RTFM (Read to Fight Monsters), a novel RL environment where agents must read a text document to understand changing world dynamics. Alongside this environment, they introduce $txt2\pi$, a model utilizing bidirectional feature modulation that successfully generalizes to completely novel environments by "reading" how to play rather than merely memorizing static patterns.
The Challenge of Dynamic Environments
Most "language-conditioned" RL focuses on using text to specify goals (e.g., "Go to the red room"). However, in the real world, language is frequently used to understand dynamics. We read manuals, wikis, or instructions to learn how a system works before we interact with it.
The authors identify a critical gap in current RL capabilities: agents need to perform "multi-hop reasoning" across multiple sources of information. To succeed in ever-changing environments, an agent must synthesize a goal ("Defeat the enemy team"), a manual ("The Rebel Enclave consists of wolves..."), and its current visual observations ("I see a wolf and a fire sword").
The RTFM Environment
To test this capability, the paper introduces RTFM (Read to Fight Monsters), a procedurally generated grid-world environment.
- The Setup: The world consists of a grid populated with monsters, teams, elements (e.g., fire, poison), and items with specific modifiers (e.g., fanatical, arcane).
- The Challenge: The rules change in every episode. A "wolf" might belong to the "Order of the Forest" team in one game and the "Star Alliance" in the next. Similarly, "arcane" items might defeat "fire" monsters in one instance, but "poison" monsters later.
- Complexity: Through random templates and rule shuffling, the environment can generate over 15 billion unique documents. Winning requires a 7-step reasoning chain, from identifying the target team in the manual to picking up the specific item that counters the specific monster currently present on the board.
Methods: The $txt2\pi$ Architecture
To solve RTFM, the authors propose a novel architecture called $txt2\pi$. The core engine of this model is the $FiLM^2$ (Bidirectional Feature-wise Linear Modulation) layer.
While the original FiLM architecture uses text to influence how a neural network processes images, $FiLM^2$ establishes a two-way street:
- Text-to-Visual: Language features modulate the visual representation (e.g., instructing the visual system to "Look for the wolf mentioned in the text").
- Visual-to-Text: Visual observations modulate the text representation (e.g., instructing the reading system to "Focus on the part of the manual describing the items I can actually see right now").
The model uses BiLSTMs to encode the goal and the document, processing the grid-world environment as a set of feature maps. By stacking these $FiLM^2$ layers, the agent performs multiple "hops" of reasoning, effectively cross-referencing the manual against the map.
Key Insights and Results
- Reading as Generalization: The core insight is that language is a powerful tool for generalization. If an agent learns how to read the relationship between entities, it can handle entirely new scenarios. It doesn't matter if it has never seen a "lightning-type goblin" before; it can simply look up the counter-item in the manual.
- Bidirectional Processing is Crucial: Simply using text to filter visual features (standard FiLM) proved insufficient. The agent performs significantly better when the visual state also helps it "focus" its reading on relevant parts of the document.
- Superior Generalization: On the simplest RTFM tasks, $txt2\pi$ achieved an 84% win rate, compared to just 49% for standard FiLM and 24% for a language-conditioned CNN.
- Zero-Shot Dynamics: When tested on environments with dynamics (monster-team-element assignments) never seen during training, $txt2\pi$ maintained its performance. This proves the agent was actually reading the document rather than memorizing a lookup table.
- Curriculum Learning: The full RTFM task is too complex to learn from scratch. By "onboarding" agents through simpler versions (smaller maps, stationary monsters), the model eventually achieved a 61% win rate on the most complex version of the task (10x10 maps with moving monsters and natural language templates).
Limitations and Future Work
Despite the impressive success of $txt2\pi$, it still trails human performance. Humans can solve these puzzles almost 100% of the time due to a superior grasp of coreference and logical deduction.
Furthermore, while the model handles templated natural language well, it has not yet been tested on truly "wild" text, such as a real video game wiki. Future research directions include scaling to longer, more diverse documents and exploring hierarchical policies where an agent explicitly "plans" its moves based on the evidence extracted from the text.
Summary updated