Abstract
Creating AI agents capable of nuanced communication and goal-directed action in complex environments remains a significant challenge. This paper introduces a procedural generation pipeline designed to train agents in text-based fantasy worlds. By constructing a "curriculum of rarity"—a sequence of training tasks that focuses on the long tail of human-generated data—we demonstrate that agents can learn more robust, generalizable representations. Our results show a significant improvement in zero-shot goal achievement compared to standard training methods.
Introduction
In the pursuit of human-like AI, "situated" environments such as text-adventure games serve as a critical benchmark. In these settings, agents must interpret textual descriptions, reason about complex motivations, and interact with other entities to achieve specific goals. However, Reinforcement Learning (RL) agents often suffer from a "generalization gap," where they memorize specific training trajectories rather than learning underlying logic.
This issue is compounded by the skewed nature of human-created datasets. In the LIGHT fantasy framework, common actions like "get" or "hit" dominate the data, leaving agents ill-equipped for rarer, more complex interactions. To address this, we propose a method to procedurally generate novel training environments that emphasize these rare occurrences.
Methodology: Procedural Environment Generation
Our approach utilizes a three-stage pipeline to create a diverse and coherent training curriculum.
1. World and Quest Generation
The system selects an initial character and location, then employs a generative language model (BART) to produce a "short motivation" and a corresponding "goal action." For example, a dragon might be motivated to recover a stolen egg, leading to the goal: "take egg, hit knight."
2. Alignment and Coherence
To ensure the generated quests are playable and believable, we use BERT-based models to "flesh out" the environment. This alignment step retrieves contextually relevant characters, objects, and neighboring locations. This prevents "spurious diversity"—the generation of nonsensical worlds (e.g., a castle in a desert) that can degrade agent learning.
3. The Curriculum of Rarity
Difficulty is quantified by the rarity of quest components in the original dataset. We generate pools of quests with increasingly flat distributions, forcing the agent to master rare verbs and nouns. Agents are trained using an Actor-Critic (A2C) framework, receiving rewards for both physical goal completion ("Act Goals") and persona-consistent dialogue ("Speech Goals").
Experimental Results
We evaluated the agents on a zero-shot goal achievement task using 211 human-expert demonstrations.
- Baseline Performance: Agents trained solely on the original LIGHT-Quests data achieved a score of 0.303 for all goals.
- Procedural Curriculum: The introduction of the generated curriculum raised this score to 0.373.
- Action-Specific Success: In terms of "Act Goals," the best-performing model (utilizing an adaptive encoder and the procedural curriculum) reached 0.506, a substantial improvement over the 0.420 baseline.
Ablation studies confirmed that generating new rare quests is more effective than simply oversampling existing ones, highlighting the importance of environmental diversity.
Discussion and Limitations
While the procedural curriculum significantly enhances generalization, challenges remain. The current "achievability rate" of generated quests is approximately 89%, with some quests remaining impossible due to alignment errors. Furthermore, the scale of generated worlds is currently limited to small clusters of locations and characters. Future work will focus on scaling these environments and increasing the complexity of multi-turn social interactions and negotiations.
Conclusion
This research demonstrates that procedural generation, when guided by a curriculum of rarity, is a powerful tool for bridging the generalization gap in situated dialogue agents. By focusing on the "long tail" of interaction data, we can move closer to agents that truly understand the logic of their environments and the nuances of human communication.
Summary updated