Abstract
Text-based games present a unique challenge for reinforcement learning due to their vast combinatorial action spaces and inherent partial observability. This paper presents a "skill-centric" framework that enables agents to autonomously abstract complex action sequences into high-level, reusable skills. By integrating the Information Bottleneck principle with self-predictive dynamics, the agent learns to navigate environments more efficiently. Our framework, evaluated on the Jericho benchmark, demonstrates significant performance gains over prior state-of-the-art methods, particularly in classic titles such as Zork1.
Introduction
Classic text-based adventure games, such as Zork, have long served as a rigorous testing ground for artificial intelligence. Unlike environments with discrete, low-dimensional action spaces (e.g., Chess or Atari), text games require agents to interact using natural language. This leads to two primary difficulties:
- Combinatorial Action Space: The number of possible text commands at any given state is astronomical. For instance, in Zork1, the action space can reach approximately $1.64 \times 10^{14}$ possible combinations.
- Partial Observability: Agents receive information only through textual descriptions of their immediate surroundings, lacking a global state view. This makes long-term planning and state tracking exceptionally difficult.
While previous research often relied on human-curated walkthroughs or predefined sub-goals, this work proposes an autonomous framework for skill discovery and representation learning.
The "Abstract then Play" Framework
The core of our approach is the "Abstract then Play" framework, which builds upon a Deep Q-Network (DQN) architecture. The framework focuses on learning a latent skill variable $z$ that represents a sequence of primitive actions, effectively creating "macro-actions" that simplify the search space.
Skill Learning via Information Bottleneck
To ensure that learned skills are both meaningful and distinct, we apply the Information Bottleneck (IB) principle. The objective is to maximize the mutual information between the latent skill $z$ and the resulting trajectory, while minimizing the inclusion of irrelevant "noise." This encourages the agent to discover skills that are highly informative for reaching specific goals. To prevent the collapse of skill representations into a single mode, we utilize a discriminator to distribute skill embeddings uniformly across a unit hypersphere.
Self-Predictive Dynamics
The framework incorporates three internal mechanisms to refine the agent's understanding of the environment and its own actions:
- Inverse Dynamics: The agent learns to predict the action taken by observing the transition between two consecutive states ($s_t$ and $s_{t+1}$).
- Forward Dynamics: The agent predicts the subsequent state $s_{t+1}$ given the current state $s_t$ and the chosen action $a_t$.
- Self-Recovery: To ensure high-quality embeddings, a reconstruction loss is used to recover the original text action from its mathematical representation.
Experimental Setup
We evaluated the framework using the Jericho environment, a standard benchmark for interactive fiction. Jericho provides a template-based interface for 15 classic games of varying complexity. Our experiments compared the "Abstract then Play" framework against several strong baselines, including Deep Reinforcement Relevance Network (DRRN), CALM, and Case-Based Reasoning (CBR) agents.
Results and Analysis
Our framework achieved state-of-the-art performance across the majority of the tested games. Notably, in Zork1, our agent reached a score of 51, surpassing the previous best of 44.3 held by the CBR model. Similarly, in Inhumane, it achieved a score of 28.7, outperforming CALM's 25.7. In games like Karn and Yomomma, the agent matched or exceeded the performance of all previous specialized models.
Key Insights
- State Differences: Ablation studies revealed that the difference between states ($s_{t+1} - s_t$) is a critical feature for skill extraction, as it captures the fundamental changes in the environment caused by an action more effectively than the state or action alone.
- Embedding Stability: Continuous skill embeddings were found to be more stable and effective than discrete alternatives, better capturing the proximity and relationships between similar behaviors.
Conclusion and Future Work
The "Abstract then Play" framework represents a significant advancement in autonomous skill discovery for text-based RL. By abstracting the action space, agents can overcome the limitations of natural language interaction. Future research will focus on addressing sparse reward environments through enhanced curiosity-driven exploration and developing more flexible, goal-directed skill sampling methods to tackle even longer-horizon puzzles.
Summary updated