Abstract
Reinforcement Learning (RL) agents in text-based games (TBGs) often struggle with generalization, frequently memorizing specific "flavor text" rather than learning underlying game logic. This paper introduces CREST (Context Relevant Episodic State Truncation), a two-stage bootstrapping method that prunes irrelevant textual artifacts from game observations. By measuring semantic similarity between observation tokens and successful action tokens, CREST creates an information bottleneck that forces agents to focus on task-relevant information. Experimental results on the TextWorld framework demonstrate that CREST achieves state-of-the-art generalization, requiring up to 20x less training data than standard models.
The Challenge of "Flavor Text" in TBGs
In text-based environments like Zork or TextWorld, the state is described entirely through natural language. These descriptions often include irrelevant sentences—such as "You don't like doors? Why not try going west?"—where the only critical information is the existence of an exit to the west. Standard RL agents, such as LSTM-based Deep Recurrent Q-Networks (DRQNs), are prone to overfitting on this noise. Instead of learning general navigational concepts, they associate specific, irrelevant phrases with rewards, leading to failure when moved to new games with different descriptions.
The CREST Methodology
To combat this lack of generalization, the authors propose a two-stage "bootstrapping" approach designed to filter out environmental noise:
- Base Model Training: A standard Q-learning agent is trained on the original, noisy text. This model is allowed to overfit until it can solve the training games perfectly.
- Action Token Extraction: The system aggregates all action tokens (verbs and nouns like "go," "east," "coin") used by the overfitted base model during successful episodes.
- CREST Pruning: For every observation, a Token Relevance Distribution (TRD) is calculated. Using semantic embeddings (such as ConceptNet), the system measures the similarity between each word in the observation and the successful action tokens. Words falling below a specific threshold are pruned.
- Bootstrapped Model Training: A second agent is trained from scratch using only the pruned, high-relevance text.
Semantic Knowledge and Pruning
A key insight of the paper is the importance of the semantic knowledge source. The researchers found that ConceptNet, a common-sense knowledge graph, outperformed standard word embeddings like Word2Vec. ConceptNet’s ability to recognize that a "fridge" is related to "food" or "kitchen" allows the agent to identify relevant objects even if the exact word never appeared in previous successful actions.
By creating a "hard attention mask" through pruning, the authors implement an information bottleneck. This prevents the agent from even perceiving the spurious correlations that typically lead to memorization and overfitting.
Key Results and Generalization
The CREST-based agent demonstrated significant improvements in both efficiency and performance:
- Data Efficiency: Agents trained on only 25–50 games achieved performance comparable to or better than state-of-the-art models trained on 500 games, representing a 20x reduction in required training data.
- Faster Convergence: The bootstrapped models reached peak performance in significantly fewer training episodes than baselines.
- Zero-Shot Transfer: In tests where agents were trained on short quests (15 rooms) and tested on long quests (25 rooms), the CREST agent significantly outperformed standard DRQNs. This indicates that the agent learned the logic of navigation rather than just memorizing specific paths.
- Handling Complexity: On "Hard" levels of the CoinCollector game, the proposed method achieved success rates between 0.88 and 0.93, while standard LSTM-DRQNs struggled to exceed 0.33.
Conclusion and Future Work
CREST demonstrates that overfitting can be leveraged as a tool for feature selection. By using an overfitted model to identify relevant tokens, researchers can train more robust, generalized agents. Future work aims to address scenarios where there is minimal semantic overlap between training and testing domains, as well as environments where goal statements change significantly between games.
Summary updated