Abstract
Reinforcement Learning (RL) methods for solving Text-Based Games (TBGs) often fail to generalize on unseen games, especially in small data regimes. To address this issue, we propose Context Relevant Episodic State Truncation (CREST) for irrelevant token removal in observation text for improved generalization. Our method first trains a base model using Q-learning, which typically overfits the training games. The base model's action token distribution is used to perform observation pruning that removes irrelevant tokens. A second bootstrapped model is then retrained on the pruned observation text. Our bootstrapped agent shows improved generalization in solving unseen TextWorld games, using 10x-20x fewer training games compared to previous state-of-the-art methods despite requiring fewer training episodes.
Introduction
Text-Based Games (TBGs) like Zork or TextWorld present a unique challenge for AI because they are partially observable environments where the state is communicated entirely through natural language. While modern RL agents can learn to solve specific game instances, they frequently fail to generalize to unseen games.
The primary hurdle is overfitting to "flavor text"—irrelevant descriptive noise that does not impact game logic (e.g., "You don't like doors? Why not try going west?"). Standard agents often memorize these specific strings as cues for actions. When encountered in a new game with different flavor text, these memorized strategies collapse. CREST addresses this by filtering out non-essential text, forcing the agent to learn functional relationships rather than surface-level patterns.
Methodology: The CREST Framework
The authors propose a two-stage bootstrapping approach designed to isolate essential game state information from linguistic noise.
- Base Model Training: A standard RL agent (LSTM-DRQN with attention) is trained on the original, noisy text. While this model overfits, it successfully identifies the vocabulary of effective actions (verbs and nouns).
- Episodic Action Token Aggregation (EATA): The system aggregates all action tokens (e.g., "go," "north," "coin") used by the base model during successful episodes.
- Observation Pruning: For each game step, every word in the observation text is compared to the EATA list. A Token Relevance Distribution (TRD) is calculated using semantic similarity. The researchers found that ConceptNet embeddings were particularly effective here, as they capture commonsense functional relationships (e.g., relating "fridge" to "kitchen") better than statistical models like Word2Vec.
- Bootstrapped Training: A second agent is trained from scratch on the "pruned" text, which contains only tokens that exceed a relevance threshold. This "cleaned" environment prevents the agent from relying on irrelevant linguistic cues.
Experimental Evaluation
The study utilized the TextWorld Coin-collector framework across Easy, Medium, and Hard levels. Training was conducted on quests requiring 15 rooms (L15), with testing performed on unseen games and longer quests (L20, L25) to evaluate zero-shot transfer. A secondary test using Cooking Games evaluated the model's ability to handle unseen objects, such as substituting "cilantro" for "parsley."
Key Results and Insights
CREST demonstrates significant improvements in both data efficiency and generalization capabilities over standard DQN and DRQN baselines.
- Data Efficiency: CREST achieved performance with only 25 training games that standard models could not match even with 500 games—representing a 20x improvement in data efficiency.
- Generalization: On "Hard" mode unseen games, CREST maintained a success rate of approximately 0.88, whereas baseline models dropped to near zero. It also showed robust zero-shot transfer when moved from 15-room quests to 25-room quests.
- Action-Centric Filtering: The core insight is that an agent's own successful history is the most effective filter for environmental relevance. By "blinding" the agent to irrelevant state space, semantic pruning acts as a powerful regularizer.
Conclusion
CREST provides a computationally efficient alternative to complex knowledge graphs for solving TBGs. By leveraging semantic similarity as a regularization tool, it enables agents to focus on the functional mechanics of a game. Future work will explore applying this method to games with larger domain differences and handling novel objects that lack semantic neighbors in the training set.
Summary updated