Abstract
A hallmark of human intelligence is the ability to understand and communicate with language. Interactive Fiction games are fully text-based simulation environments where a player issues text commands to effect change in the environment and progress through the story. We argue that IF games are an excellent testbed for studying language-based autonomous agents. In particular, IF games combine challenges of combinatorial action spaces, language understanding, and commonsense reasoning. To facilitate rapid development of language-based agents, we introduce Jericho, a learning environment for man-made IF games and conduct a comprehensive study of text-agents across a rich set of games, highlighting directions in which agents can improve.
Introduction
While reinforcement learning (RL) has achieved remarkable success in visual domains like Atari and Go, language-based RL remains a "colossal" challenge. Interactive Fiction (IF) games, such as the classic Zork, require agents to process rich text descriptions and respond with meaningful text commands. This paper introduces Jericho, a new learning environment that enables RL agents to play human-made text adventure games, serving as a formidable testbed for natural language understanding and commonsense reasoning.
The Challenges of Text-Based RL
The authors identify three primary hurdles that make IF games difficult for traditional AI:
- Combinatorial Action Space: A simple four-word command drawn from a modest vocabulary can result in billions of possible combinations, most of which are nonsensical to the game's parser.
- Commonsense Reasoning: Humans intuitively know that a "locked chest" requires a "key." Agents lack this embodied experience and struggle to prioritize meaningful interactions over random ones.
- Knowledge Representation: Navigating complex, non-Euclidean maps and tracking nested objects (e.g., a key inside a box inside a room) requires sophisticated mental mapping.
The Jericho Environment
Jericho is an open-source Python environment, compatible with OpenAI Gym, that interfaces with the Z-Machine (the engine behind classic Infocom games). To make these games tractable, Jericho offers several innovations:
- World Object Tree: Extracts the game's internal state as a tree of objects and locations, providing a semi-interpretable representation.
- Valid Action Identification: A "handicap" mode identifies which actions will actually change the world state, helping agents bypass "parser frustration."
- Template-Based Action Generation: Instead of generating raw strings, agents select a "template" (e.g.,
put OBJ in OBJ) and fill in the blanks with vocabulary words.
Agent Architectures
The study evaluates three types of agents: the Deep Reinforcement Relevance Network (DRRN), a choice-based agent; the Template-DQN (TDQN), a parser-based agent; and NAIL, a heuristic-based baseline. As shown in Figure 2, the TDQN architecture specifically addresses the action space challenge by learning to predict both templates and their parameters.
Experimental Results
The authors conducted a comprehensive study across 32 human-made IF games, ranging from classic dungeon crawls to complex narrative horror. This represents a significant increase in complexity over previous synthetic benchmarks.
Key insights from the results include:
- RL Viability: Both DRRN and TDQN made progress in many games, though overall completion rates remain low (average scores: NAIL 4.9%, TDQN 6.1%, DRRN 1.7%).
- Specificity vs. Generality: DRRN performed exceptionally well on games with focused action spaces, like Detective, by concentrating on high-value navigational actions.
- The "Moonshot" Games: Several titles, such as Anchorhead, remained completely unsolved (0% score), highlighting challenges with long-term dependencies and sparse rewards.
Conclusion and Future Work
Jericho demonstrates that human-made IF games are a rich environment for developing language-aware AI. While current agents are "single-game" experts, future research will focus on generalization—training agents across hundreds of games to acquire "common sense." The authors suggest that Transformer-based architectures and unsupervised learning will be essential for capturing the nuanced prose and creative action generation required to master these digital worlds.
Summary updated