Abstract
Text-based games present a significant challenge for reinforcement learning due to their combinatorial action spaces and sparse rewards. We present a two-stage framework that decouples exploration from generalization. First, a modified Go-Explore algorithm identifies high-reward trajectories. Second, a Seq2Seq model is trained via imitation learning on these trajectories. Our approach achieves state-of-the-art performance on the CookingWorld benchmark and demonstrates superior zero-shot generalization to unseen games compared to traditional reinforcement learning (RL) baselines.
Introduction
Text-based games, such as the classic Zork, represent a unique frontier for artificial intelligence, requiring a combination of natural language understanding, multi-step reasoning, and language generation. The primary obstacle in these environments is the combinatorial action space. An agent may need to generate a command of up to seven words from a 20,000-word vocabulary, resulting in approximately $10^{30}$ possible actions at any given turn. This "needle in a haystack" problem renders standard RL exploration techniques, such as random $\epsilon$-greedy, largely ineffective.
While existing solutions often simplify the task by restricting agents to a small set of pre-defined "admissible actions" provided by the game engine, this research aims to develop agents capable of operating in unconstrained natural language spaces and generalizing their skills to previously unseen environments.
Methodology
The proposed framework addresses the limitations of joint exploration and learning by splitting the process into two distinct phases:
Phase 1: Exploration via Go-Explore
The agent utilizes a modified version of the Go-Explore algorithm to discover winning trajectories. Unlike the original version designed for pixel-based environments, this variant represents game states as "cells" by summing word embeddings of room descriptions and concatenating them with the current cumulative reward. This "reward-as-state" innovation prevents the agent from conflating identical physical locations at different stages of a quest. The agent maintains an archive of these cells, "jumps" back to promising states, and explores randomly to find higher-reward trajectories.
Phase 2: Generalization via Imitation Learning
Once high-reward trajectories are archived, they are used as a dataset to train a Seq2Seq (Encoder-Decoder LSTM) model. Rather than simply replaying paths, the model learns to map text observations to token sequences that form valid, high-reward actions. This approach allows the agent to internalize the "language of the game," facilitating generalization to new scenarios rather than rote memorization.
Experimental Setup
The framework was evaluated on two primary benchmarks:
- CoinCollector: A navigation-focused task with sparse rewards, used to measure the speed of finding winning paths in mazes.
- CookingWorld (First TextWorld Problems): A diverse suite of 4,440 games involving complex tasks like following recipes and using tools. This dataset supports evaluation in Single (individual game), Joint (multi-game training), and Zero-Shot (unseen game) settings.
Results and Discussion
The two-phase approach demonstrated significant improvements in both efficiency and performance over traditional RL baselines like DQN++ and DRQN++.
As shown in the CoinCollector results, the Go-Explore variant identified winning trajectories in approximately 205k frames, more than twice as fast as the DQN++ and DRQN++ baselines, which required over 500k frames. This efficiency is critical in environments where rewards are extremely sparse.
In the more complex CookingWorld environment, the Go-Explore + Seq2Seq model outperformed the strongest RL baseline (DRRN) across all metrics. Most notably, the model exhibited exceptional zero-shot generalization, scoring 1,038 on unseen games compared to DRRN's 451. This suggests that the imitation learning phase effectively captures generalizable "common sense" logic for interacting with text-based environments.
Conclusion and Future Work
By separating exploration from generalization, the proposed framework provides a stable and efficient method for solving text-based games. However, challenges remain. In the "Joint" training setting, the agent occasionally suffers from state aliasing, where identical descriptions in different games require different actions. Future work may involve incorporating Knowledge Graphs to disambiguate these states. Additionally, while the current model relies on admissible actions during the exploration phase, future iterations aim to tackle fully unconstrained natural language spaces and stochastic environments.
Summary updated