Abstract
ScriptWorld is a novel text-based reinforcement learning (RL) environment designed to evaluate and train agents on "script knowledge"—the procedural common sense required for everyday tasks. Unlike traditional fantasy-based text games, ScriptWorld leverages human-annotated scripts to create complex, branching scenarios for real-world chores like baking a cake or taking a bus. Our experiments demonstrate that while these tasks present significant challenges for standard RL baselines, the integration of contextual embeddings and recurrent architectures allows agents to capture and generalize procedural logic across related domains.
Introduction
Reinforcement learning in text-based environments has traditionally focused on fictional fantasy worlds, such as those found in the Jericho framework or classic games like Zork. While these environments test logic and memory, the knowledge acquired—such as navigating dungeons or using magical items—rarely translates to real-world utility.
Humans navigate daily life using "Script Knowledge": an implicit understanding of the stereotypical sequences of events involved in common activities. For instance, the task of "washing dishes" involves a series of fine-grained steps including scraping food, applying soap, scrubbing, and rinsing. Current AI agents often lack this grounded, procedural common sense. ScriptWorld addresses this gap by providing a benchmark specifically focused on daily human activities.
The ScriptWorld Environment
The authors developed ScriptWorld as a choice-based interaction model built upon the DeScript corpus, a crowd-sourced collection of Event Sequence Descriptions (ESDs).
Graph Construction
The environment is structured as "scenario graphs" where each node represents a sub-task (e.g., "add soap") and edges represent the logical flow between them. To mirror real-world flexibility, the graphs allow for parallel paths (e.g., choosing between stairs or an elevator) and linguistic variations in task descriptions.
Dynamic Interaction
At each step, an agent is presented with a list of textual choices. These include a correct action that advances the task and several "distractors" sampled from other parts of the script or unrelated actions. This setup requires the agent to not only identify the next logical step but also to distinguish it from contextually similar but temporally incorrect actions.
Methods and Baselines
To establish performance benchmarks, the researchers evaluated several standard RL algorithms, including DQN, A2C, PPO, and Recurrent PPO (RPPO).
A critical component of the agent architecture is the use of pre-trained SBERT (Sentence-BERT) models. These models convert textual descriptions into dense vector embeddings, providing the RL agents with a rich, contextual representation of the state space. Additionally, a "handicap" system using GPT-2 was implemented to provide textual hints, helping to calibrate the difficulty of the environment for learning agents.
Key Results and Insights
The benchmark includes 10 daily activities with vast state spaces; for example, the "Flying in an Airplane" scenario contains over $10^{30}$ possible correct paths.
Algorithm Performance
Recurrent PPO (RPPO) emerged as the most effective baseline. Its success is attributed to its recurrent (LSTM) architecture, which enables the agent to maintain a memory of completed sub-tasks within long, sequential procedures.
Contextual Understanding and Generalization
The study found that contextual embeddings (SBERT) significantly outperform static word vectors like GloVe. In procedural tasks, the meaning of a word is highly dependent on its temporal and task-specific context. Furthermore, agents demonstrated "procedural generalization." An agent trained on the "Ride a Bus" scenario performed significantly better than a random baseline when introduced to a "Go on a Train" scenario, suggesting that the models are learning the underlying logic of public transportation rather than merely memorizing text strings.
Conclusion
ScriptWorld provides a vital bridge between text-based RL and real-world procedural knowledge. While the current benchmark is limited to 10 high-quality scenarios, it highlights the complexity of learning human procedures from scratch. Future work will focus on automating script alignment to expand the environment to hundreds of activities and transitioning from choice-based to parser-based (free-form text) interactions to further challenge agent capabilities.
Summary updated