Abstract
We teach goal-driven agents to interactively act and speak in situated environments by training on generated curriculums. Our agents operate in LIGHT (Urbanek et al. 2019) -- a large-scale crowd-sourced fantasy text adventure game wherein an agent perceives and interacts with the world through textual natural language. Goals in this environment take the form of character-based quests, consisting of personas and motivations. We augment LIGHT by learning to procedurally generate additional novel textual worlds and quests to create a curriculum of steadily increasing difficulty for training agents to achieve such goals. In particular, we measure curriculum difficulty in terms of the rarity of the quest in the original training distribution -- an easier environment is one that is more likely to have been found in the unaugmented dataset. An ablation study shows that this method of learning from the tail of a distribution results in significantly higher generalization abilities as measured by zero-shot performance on never-before-seen quests.
Introduction
In the domain of situated dialogue, agents must navigate complex environments using both physical actions and natural language to achieve specific goals. However, current reinforcement learning (RL) agents often suffer from a significant generalization gap. Rather than learning the underlying logic of a task, these agents tend to "memorize trajectories"—scripts that work in specific, common scenarios but fail when environmental variables change. This behavior is largely driven by "head-heavy" training distributions dominated by a few common interaction types, leaving agents unprepared for the "long tail" of rare, complex scenarios.
Procedural Environment Generation (PEG)
To address this, we introduce a Procedural Environment Generation (PEG) pipeline designed to create a training curriculum that forces agents to learn broader, more robust skills.
The pipeline consists of several key stages:
- World and Quest Generation: The system selects an initial character and location, then utilizes a generative language model (BART) to synthesize a unique motivation and goal.
- Alignment: To ensure playability, BERT-based bi-encoders are used to populate the world with relevant entities. If a quest involves a "knight" and an "egg," the alignment module ensures these are placed in logically connected locations.
- Curriculum Design: Difficulty is parameterized by the rarity of the quest's primary verb. The curriculum begins with the original skewed data and gradually introduces rarer, procedurally generated quests to achieve a near-uniform task distribution.
Agent Architecture
Agents are trained using the Advantage Actor-Critic (A2C) framework. We evaluate two primary encoder types: a "Scratch" encoder and an "Adaptive" encoder pre-trained on large-scale datasets like Reddit and ATOMIC-LIGHT.
The agent perceives the setting, its persona/motivation, and the full action/dialogue history. This information is encoded and passed to action and dialogue policy networks, which interact with the game engine and a "Dungeon Master" (DM) model that provides rewards for natural speech.
Datasets and Pre-training
Our primary environment is LIGHT, a large-scale fantasy text adventure framework. We utilize LIGHT-Quests, which adds goal-oriented motivations. For the "Adaptive" models, pre-training on Reddit provides a foundation for natural dialogue, while ATOMIC-LIGHT offers commonsense knowledge about actions and consequences.
Key Results and Insights
The agents were evaluated on zero-shot performance across 211 human-demonstrated quests.
- Generalization Boost: The procedural curriculum significantly improved success rates. For the "Adaptive" model, the success rate rose from 0.303 to 0.373.
- Pre-training Synergy: Pre-trained models benefited more from the procedural curriculum, suggesting that foundational commonsense knowledge is crucial for learning from diverse, generated environments.
- Coherence Requirements: Randomly generated, nonsensical worlds led to poor performance, highlighting the necessity of internal consistency for effective situated learning.
Limitations and Future Work
While promising, the PEG pipeline faces challenges. Approximately 11% of generated quests were unachievable due to alignment errors. Additionally, the current scale of generated worlds is limited. Future work will focus on scaling to larger environments, improving quest achievability, and refining the reward mechanisms for dialogue.
Summary updated