Exploration Based Language Learning for Text-Based Games

Andrea Madotto, Mahdi Namazifar, Joost Huizinga, Piero Molino, Adrien Ecoffet, Huaixiu Zheng, Alexandros Papangelis, Dian Yu, Chandra Khatri, Gökhan Tür

Abstract

Text-based games present a significant challenge for reinforcement learning due to their combinatorial action spaces and sparse rewards. We present a two-stage framework that decouples exploration from generalization. First, a modified Go-Explore algorithm identifies high-reward trajectories. Second, a Seq2Seq model is trained via imitation learning on these trajectories. Our approach achieves state-of-the-art performance on the CookingWorld benchmark and demonstrates superior zero-shot generalization to unseen games compared to traditional reinforcement learning (RL) baselines.

Introduction

Text-based games, such as the classic Zork, represent a unique frontier for artificial intelligence, requiring a combination of natural language understanding, multi-step reasoning, and language generation. The primary obstacle in these environments is the combinatorial action space. An agent may need to generate a command of up to seven words from a 20,000-word vocabulary, resulting in approximately $10^{30}$ possible actions at any given turn. This "needle in a haystack" problem renders standard RL exploration techniques, such as random $\epsilon$-greedy, largely ineffective.

While existing solutions often simplify the task by restricting agents to a small set of pre-defined "admissible actions" provided by the game engine, this research aims to develop agents capable of operating in unconstrained natural language spaces and generalizing their skills to previously unseen environments.

Methodology

The proposed framework addresses the limitations of joint exploration and learning by splitting the process into two distinct phases:

Phase 1: Exploration via Go-Explore

The agent utilizes a modified version of the Go-Explore algorithm to discover winning trajectories. Unlike the original version designed for pixel-based environments, this variant represents game states as "cells" by summing word embeddings of room descriptions and concatenating them with the current cumulative reward. This "reward-as-state" innovation prevents the agent from conflating identical physical locations at different stages of a quest. The agent maintains an archive of these cells, "jumps" back to promising states, and explores randomly to find higher-reward trajectories.

Phase 2: Generalization via Imitation Learning

Once high-reward trajectories are archived, they are used as a dataset to train a Seq2Seq (Encoder-Decoder LSTM) model. Rather than simply replaying paths, the model learns to map text observations to token sequences that form valid, high-reward actions. This approach allows the agent to internalize the "language of the game," facilitating generalization to new scenarios rather than rote memorization.

Experimental Setup

The framework was evaluated on two primary benchmarks:

  1. CoinCollector: A navigation-focused task with sparse rewards, used to measure the speed of finding winning paths in mazes.
  2. CookingWorld (First TextWorld Problems): A diverse suite of 4,440 games involving complex tasks like following recipes and using tools. This dataset supports evaluation in Single (individual game), Joint (multi-game training), and Zero-Shot (unseen game) settings.

Results and Discussion

The two-phase approach demonstrated significant improvements in both efficiency and performance over traditional RL baselines like DQN++ and DRQN++.

Comparison of learning efficiency on the Coin Collector task.

As shown in the CoinCollector results, the Go-Explore variant identified winning trajectories in approximately 205k frames, more than twice as fast as the DQN++ and DRQN++ baselines, which required over 500k frames. This efficiency is critical in environments where rewards are extremely sparse.

In the more complex CookingWorld environment, the Go-Explore + Seq2Seq model outperformed the strongest RL baseline (DRRN) across all metrics. Most notably, the model exhibited exceptional zero-shot generalization, scoring 1,038 on unseen games compared to DRRN's 451. This suggests that the imitation learning phase effectively captures generalizable "common sense" logic for interacting with text-based environments.

Performance comparison across Single, Joint, and Zero-Shot settings in CookingWorld.

Conclusion and Future Work

By separating exploration from generalization, the proposed framework provides a stable and efficient method for solving text-based games. However, challenges remain. In the "Joint" training setting, the agent occasionally suffers from state aliasing, where identical descriptions in different games require different actions. Future work may involve incorporating Knowledge Graphs to disambiguate these states. Additionally, while the current model relies on admissible actions during the exploration phase, future iterations aim to tackle fully unconstrained natural language spaces and stochastic environments.

Summary updated

Abstract

This work presents an exploration and imitation-learning-based agent capable of state-of-the-art performance in playing text-based computer games. Text-based computer games describe their world to the player through natural language and expect the player to interact with the game using text. These games are of interest as they can be seen as a testbed for language understanding, problem-solving, and language generation by artificial agents. Moreover, they provide a learning environment in which these skills can be acquired through interactions with an environment rather than using fixed corpora. One aspect that makes these games particularly challenging for learning agents is the combinatorially large action space. Existing methods for solving text-based games are limited to games that are either very simple or have an action space restricted to a predetermined set of admissible actions. In this work, we propose to use the exploration approach of Go-Explore for solving text-based games. More specifically, in an initial exploration phase, we first extract trajectories with high rewards, after which we train a policy to solve the game by imitating these trajectories. Our experiments show that this approach outperforms existing solutions in solving text-based games, and it is more sample efficient in terms of the number of interactions with the environment. Moreover, we show that the learned policy can generalize better than existing solutions to unseen games without using any restriction on the action space.
BibTeX
@inproceedings{madotto2020exploration,
  author = {Andrea Madotto and Mahdi Namazifar and Joost Huizinga and Piero Molino and Adrien Ecoffet and Huaixiu Zheng and Alexandros Papangelis and Dian Yu and Chandra Khatri and Gökhan Tür},
  title = {{Exploration Based Language Learning for Text-Based Games}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/207},
}
Figure