Abstract then Play: A Skill-centric Reinforcement Learning Framework for Text-based Games

Anjie Zhu, Peng-Fei Zhang, Yi Zhang, Zi Huang, Jie Shao

Abstract

Text-based games present a unique challenge for reinforcement learning due to their vast combinatorial action spaces and inherent partial observability. This paper presents a "skill-centric" framework that enables agents to autonomously abstract complex action sequences into high-level, reusable skills. By integrating the Information Bottleneck principle with self-predictive dynamics, the agent learns to navigate environments more efficiently. Our framework, evaluated on the Jericho benchmark, demonstrates significant performance gains over prior state-of-the-art methods, particularly in classic titles such as Zork1.

Introduction

Classic text-based adventure games, such as Zork, have long served as a rigorous testing ground for artificial intelligence. Unlike environments with discrete, low-dimensional action spaces (e.g., Chess or Atari), text games require agents to interact using natural language. This leads to two primary difficulties:

  1. Combinatorial Action Space: The number of possible text commands at any given state is astronomical. For instance, in Zork1, the action space can reach approximately $1.64 \times 10^{14}$ possible combinations.
  2. Partial Observability: Agents receive information only through textual descriptions of their immediate surroundings, lacking a global state view. This makes long-term planning and state tracking exceptionally difficult.

While previous research often relied on human-curated walkthroughs or predefined sub-goals, this work proposes an autonomous framework for skill discovery and representation learning.

The "Abstract then Play" Framework

The core of our approach is the "Abstract then Play" framework, which builds upon a Deep Q-Network (DQN) architecture. The framework focuses on learning a latent skill variable $z$ that represents a sequence of primitive actions, effectively creating "macro-actions" that simplify the search space.

High-level architecture of the skill-centric reinforcement learning framework.

Skill Learning via Information Bottleneck

To ensure that learned skills are both meaningful and distinct, we apply the Information Bottleneck (IB) principle. The objective is to maximize the mutual information between the latent skill $z$ and the resulting trajectory, while minimizing the inclusion of irrelevant "noise." This encourages the agent to discover skills that are highly informative for reaching specific goals. To prevent the collapse of skill representations into a single mode, we utilize a discriminator to distribute skill embeddings uniformly across a unit hypersphere.

Self-Predictive Dynamics

The framework incorporates three internal mechanisms to refine the agent's understanding of the environment and its own actions:

  • Inverse Dynamics: The agent learns to predict the action taken by observing the transition between two consecutive states ($s_t$ and $s_{t+1}$).
  • Forward Dynamics: The agent predicts the subsequent state $s_{t+1}$ given the current state $s_t$ and the chosen action $a_t$.
  • Self-Recovery: To ensure high-quality embeddings, a reconstruction loss is used to recover the original text action from its mathematical representation.

Experimental Setup

We evaluated the framework using the Jericho environment, a standard benchmark for interactive fiction. Jericho provides a template-based interface for 15 classic games of varying complexity. Our experiments compared the "Abstract then Play" framework against several strong baselines, including Deep Reinforcement Relevance Network (DRRN), CALM, and Case-Based Reasoning (CBR) agents.

Results and Analysis

Our framework achieved state-of-the-art performance across the majority of the tested games. Notably, in Zork1, our agent reached a score of 51, surpassing the previous best of 44.3 held by the CBR model. Similarly, in Inhumane, it achieved a score of 28.7, outperforming CALM's 25.7. In games like Karn and Yomomma, the agent matched or exceeded the performance of all previous specialized models.

Performance comparison of the proposed framework against baselines across 15 Jericho games.

Key Insights

  • State Differences: Ablation studies revealed that the difference between states ($s_{t+1} - s_t$) is a critical feature for skill extraction, as it captures the fundamental changes in the environment caused by an action more effectively than the state or action alone.
  • Embedding Stability: Continuous skill embeddings were found to be more stable and effective than discrete alternatives, better capturing the proximity and relationships between similar behaviors.

Conclusion and Future Work

The "Abstract then Play" framework represents a significant advancement in autonomous skill discovery for text-based RL. By abstracting the action space, agents can overcome the limitations of natural language interaction. Future research will focus on addressing sparse reward environments through enhanced curiosity-driven exploration and developing more flexible, goal-directed skill sampling methods to tackle even longer-horizon puzzles.

Summary updated

Abstract

Text-based games present an exciting test-bed for reinforcement learning algorithms in the natural language environment. In these adventure games, an agent must learn to interact with the environment through text in order to accomplish tasks, facing large and combinational action space as well as partial observability issues. However, existing solutions fail to decompose the task and abstract the action autonomously, which either pre-specify the subtasks or pre-train on the human gameplay dataset. In this work, we introduce a novel skill-centric reinforcement learning framework, which is capable of abstracting the action in an end-to-end manner. To learn a more disentangled skill, we focus on the informativeness and distinguishability of the skill in accordance with the information bottleneck principle. Specifically, we introduce a discriminator to enable the skill to reflect the trajectory and push their representations onto the unit hypersphere to distribute uniformly. Moreover, a self-predictive mechanism is employed to learn inverse and forward dynamics, and a self-recovery mechanism is leveraged to refine the action representation, thus resulting in a more comprehensive perception of dynamics and more effective representations of textual state and action. Empirical experiments are carried out on the Jericho environment and the results validate the superiority against state-of-the-art baselines.
BibTeX
@inproceedings{zhu2023abstract,
  author = {Anjie Zhu and Peng-Fei Zhang and Yi Zhang and Zi Huang and Jie Shao},
  title = {{Abstract then Play: A Skill-centric Reinforcement Learning Framework for Text-based Games}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.836},
}
Figure