Exploration Based Language Learning for Text-Based Games

Andrea Madotto, Mahdi Namazifar, Joost Huizinga, Piero Molino, Adrien Ecoffet, Huaixiu Zheng, Alexandros Papangelis, Dian Yu, Chandra Khatri, Gökhan Tür

Abstract

This work presents an exploration and imitation-learning-based agent capable of state-of-the-art performance in playing text-based computer games. Text-based games describe their world through natural language and expect text-based interaction, serving as a testbed for language understanding and problem-solving. A major challenge is the combinatorially large action space. We propose using the exploration approach of Go-Explore to extract high-reward trajectories, followed by training a policy to solve the game by imitating these trajectories. Our experiments show that this approach outperforms existing solutions in performance and sample efficiency, and generalizes better to unseen games without restricting the action space.

Introduction

Text-based games, such as the classic Zork, represent a "grand challenge" for AI, requiring a blend of natural language understanding, common-sense reasoning, and long-term planning. The primary hurdle is the combinatorial action space. In a typical game with a vocabulary of 20,000 words, a command of just 7 words results in $20,000^7$ possible actions—an astronomical number that makes standard Reinforcement Learning (RL) exploration impossible.

Existing methods often rely on a restricted list of "admissible actions" provided by the game engine. This paper introduces an agent designed to eventually operate in the full, unconstrained space of natural language and generalize to novel environments by decoupling exploration from policy learning.

Methodology

The authors propose a two-phase framework that separates the discovery of successful strategies from the linguistic task of command generation.

Phase 1: Exploration (Go-Explore)

Instead of random exploration, the agent uses the Go-Explore algorithm. It maintains an archive of "cells" representing discretized game states, defined by word embeddings of room descriptions and the current cumulative reward. The agent selects a promising cell, "teleports" back to that state (leveraging the determinism of the game simulator), and explores from there. This prevents the agent from forgetting how to reach distant, high-reward states.

Phase 2: Generalization (Imitation Learning)

Once the archive contains high-performing trajectories, they are treated as "gold standard" demonstrations. The agent trains a Seq2Seq model (an LSTM encoder-decoder) to map text observations to the successful actions found in Phase 1. This turns a fragile exploration policy into a robust, state-conditioned policy capable of zero-shot generalization.

Experimental Setup

The approach was evaluated on two distinct benchmarks:

  • CoinCollector: A navigation task in a maze of rooms, used to measure pure exploration efficiency in sparse-reward settings.
  • CookingWorld: A complex environment with 222 difficulty levels where agents must follow recipes, collect ingredients, and process food (e.g., chopping, roasting). This tests language logic and cross-game generalization.

Results and Discussion

Sample Efficiency in Sparse Rewards

In the CoinCollector benchmark, the Go-Explore approach demonstrated superior sample efficiency. It consistently found the mathematically optimal path (30 steps), whereas RL baselines often settled for sub-optimal paths (38–42 steps).

Figure 1: Comparison of learning curves on the CoinCollector benchmark. Go-Explore reaches the maximum score significantly faster than DQN++ and DRQN++ baselines, demonstrating superior sample efficiency.

Performance and Generalization

In CookingWorld, the Go-Explore + Seq2Seq agent achieved a score of 17,507, significantly outperforming the DRRN baseline (16,075). More importantly, in "Zero-Shot" settings on entirely unseen games, the proposed model scored 1,038, while the strongest baseline (DRRN) only managed 451.

Figure 2: Performance comparison across various CookingWorld tasks. The Go-Explore based agent (grey) consistently achieves higher scores across different recipe complexities compared to LSTM-DQN and DRRN baselines.

Conclusion

By decoupling discovery from learning, this framework addresses the combinatorial explosion of action spaces in text-based games. While the current method relies on game determinism for state-resetting, it demonstrates that imitation learning from high-quality explored trajectories is a powerful path toward generalizable language-based agents. Future work will focus on resolving state ambiguity through Knowledge Graphs and moving toward truly unconstrained exploration using pre-trained language models.

Summary updated

Abstract

This work presents an exploration and imitation-learning-based agent capable of state-of-the-art performance in playing text-based computer games. Text-based computer games describe their world to the player through natural language and expect the player to interact with the game using text. These games are of interest as they can be seen as a testbed for language understanding, problem-solving, and language generation by artificial agents. Moreover, they provide a learning environment in which these skills can be acquired through interactions with an environment rather than using fixed corpora. One aspect that makes these games particularly challenging for learning agents is the combinatorially large action space. Existing methods for solving text-based games are limited to games that are either very simple or have an action space restricted to a predetermined set of admissible actions. In this work, we propose to use the exploration approach of Go-Explore for solving text-based games. More specifically, in an initial exploration phase, we first extract trajectories with high rewards, after which we train a policy to solve the game by imitating these trajectories. Our experiments show that this approach outperforms existing solutions in solving text-based games, and it is more sample efficient in terms of the number of interactions with the environment. Moreover, we show that the learned policy can generalize better than existing solutions to unseen games without using any restriction on the action space.
BibTeX
@inproceedings{madotto2020exploration,
  author = {Andrea Madotto and Mahdi Namazifar and Joost Huizinga and Piero Molino and Adrien Ecoffet and Huaixiu Zheng and Alexandros Papangelis and Dian Yu and Chandra Khatri and Gökhan Tür},
  title = {{Exploration Based Language Learning for Text-Based Games}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/207},
}
Figure