Learn What Is Possible, Then Choose What Is Best: Disentangling One-To-Many Relations in Language Through Text-based Games

Benjamin Towle, Ke ZhouEMNLP 2022 · DOI

Introduction

In natural language processing, models frequently encounter "one-to-many" scenarios where a single context can lead to numerous valid responses. For instance, in a dialogue, a simple greeting can be met with a wide variety of replies, ranging from helpful to generic or even irrelevant. Standard training approaches often treat these as "one-to-one" problems, forcing models to learn an "average" response. This often results in mediocre performance, as the most effective or "expert" actions are frequently buried within noisy datasets.

Researchers have addressed this challenge by using text-based games (such as Zork) as a controlled environment. These games require agents to identify specific "gold" actions to progress, amidst a vast sea of possible but unproductive commands. To solve this, the authors introduce PASA (Posterior Action Selection from Discrete Latent Variables), a framework designed to map the full landscape of possible behaviors before selecting the optimal one.

The PASA Methodology

PASA utilizes a teacher-student architecture to disentangle the concepts of "what is possible" and "what is best."

  1. The Teacher Model: During pre-training on large, noisy datasets, the teacher model uses discrete latent variables ($z$) to categorize different types of behavior. Instead of predicting a single action, it learns to map a context and an action to specific latent "buckets," effectively capturing the one-to-many distribution of the data.
  2. Knowledge Distillation: Once the teacher understands the range of possible actions, its knowledge is distilled into a student model. During fine-tuning on high-quality expert data, the student learns to select the "best" action by analyzing the teacher's full probability distribution rather than relying on a single hard label.
Overview of the PASA framework showing the teacher-student distillation process.

Semantically Aligning the Latent Space

The researchers explored three distinct methods for organizing the teacher's latent "buckets":

  • Latent (Vanilla): The model automatically clusters behaviors based on statistical patterns.
  • Persona: Each bucket represents a different human player from the training data, capturing stylistic diversity.
  • Intent: Actions are explicitly categorized into functional types (e.g., "navigation," "examination," or "interaction") using rule-based heuristics.

The study found that categorizing by Intent was the most effective, suggesting that functional diversity is more critical for task performance than stylistic variation in goal-oriented environments.

Experimental Setup and Datasets

The framework was evaluated using two datasets that represent the "noisy vs. expert" dichotomy:

  • ClubFloyd: A large, noisy dataset of human gameplay containing a mix of expert moves and novice errors (e.g., attempting to "eat coal").
  • Jericho Walkthroughs: A small, high-quality dataset containing only the exact actions required to maximize scores and complete the games.

Key Results

PASA demonstrated significant performance gains over existing baselines. Specifically, the Intent-based PASA model achieved a 49% empirical improvement over the previous state-of-the-art model, CALM, when evaluated on the Jericho Walkthroughs dataset.

Performance comparison of PASA variants against the CALM baseline across various text-based games.

As shown in the results, PASA excelled in "Difficult Games" like balances and ludicorp, which require complex, non-obvious interactions. In these scenarios, the distillation process allowed the student to identify expert actions as clear peaks in the teacher's probability distribution, even when those actions were rare in the initial noisy pre-training data.

Conclusion and Future Work

The PASA framework proves that disentangling possible behaviors from optimal ones is a powerful strategy for navigating noisy, one-to-many language environments. While the current model relies on a defined set of "valid actions" from game engines, future work aims to scale this to open-ended generative tasks, such as summarizing social media threads or improving dialogue systems by using community metrics (like upvotes) as proxies for "gold" rewards.

Summary updated

BibTeX
@inproceedings{towle2022learn,
  author = {Benjamin Towle and Ke Zhou},
  title = {{Learn What Is Possible, Then Choose What Is Best: Disentangling One-To-Many Relations in Language Through Text-based Games}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.364},
}
Figure