Abstract
This work presents an exploration and imitation-learning-based agent capable of state-of-the-art performance in playing text-based computer games. Text-based games describe their world through natural language and expect text-based interaction, serving as a testbed for language understanding and problem-solving. A major challenge is the combinatorially large action space. We propose using the exploration approach of Go-Explore to extract high-reward trajectories, followed by training a policy to solve the game by imitating these trajectories. Our experiments show that this approach outperforms existing solutions in performance and sample efficiency, and generalizes better to unseen games without restricting the action space.
Introduction
Text-based games, such as the classic Zork, represent a "grand challenge" for AI, requiring a blend of natural language understanding, common-sense reasoning, and long-term planning. The primary hurdle is the combinatorial action space. In a typical game with a vocabulary of 20,000 words, a command of just 7 words results in $20,000^7$ possible actions—an astronomical number that makes standard Reinforcement Learning (RL) exploration impossible.
Existing methods often rely on a restricted list of "admissible actions" provided by the game engine. This paper introduces an agent designed to eventually operate in the full, unconstrained space of natural language and generalize to novel environments by decoupling exploration from policy learning.
Methodology
The authors propose a two-phase framework that separates the discovery of successful strategies from the linguistic task of command generation.
Phase 1: Exploration (Go-Explore)
Instead of random exploration, the agent uses the Go-Explore algorithm. It maintains an archive of "cells" representing discretized game states, defined by word embeddings of room descriptions and the current cumulative reward. The agent selects a promising cell, "teleports" back to that state (leveraging the determinism of the game simulator), and explores from there. This prevents the agent from forgetting how to reach distant, high-reward states.
Phase 2: Generalization (Imitation Learning)
Once the archive contains high-performing trajectories, they are treated as "gold standard" demonstrations. The agent trains a Seq2Seq model (an LSTM encoder-decoder) to map text observations to the successful actions found in Phase 1. This turns a fragile exploration policy into a robust, state-conditioned policy capable of zero-shot generalization.
Experimental Setup
The approach was evaluated on two distinct benchmarks:
- CoinCollector: A navigation task in a maze of rooms, used to measure pure exploration efficiency in sparse-reward settings.
- CookingWorld: A complex environment with 222 difficulty levels where agents must follow recipes, collect ingredients, and process food (e.g., chopping, roasting). This tests language logic and cross-game generalization.
Results and Discussion
Sample Efficiency in Sparse Rewards
In the CoinCollector benchmark, the Go-Explore approach demonstrated superior sample efficiency. It consistently found the mathematically optimal path (30 steps), whereas RL baselines often settled for sub-optimal paths (38–42 steps).
Performance and Generalization
In CookingWorld, the Go-Explore + Seq2Seq agent achieved a score of 17,507, significantly outperforming the DRRN baseline (16,075). More importantly, in "Zero-Shot" settings on entirely unseen games, the proposed model scored 1,038, while the strongest baseline (DRRN) only managed 451.
Conclusion
By decoupling discovery from learning, this framework addresses the combinatorial explosion of action spaces in text-based games. While the current method relies on game determinism for state-resetting, it demonstrates that imitation learning from high-quality explored trajectories is a powerful path toward generalizable language-based agents. Future work will focus on resolving state ambiguity through Knowledge Graphs and moving toward truly unconstrained exploration using pre-trained language models.
Summary updated