Abstract
Artificial intelligence research in text-based games has traditionally focused on fantasy settings, which offer limited transferability to real-world tasks. We introduce ScriptWorld, a reinforcement learning (RL) environment grounded in human-written "scripts" of everyday activities such as baking a cake or catching a train. By representing these tasks as complex directed graphs, ScriptWorld provides a benchmark for evaluating an agent's procedural common sense. Our results demonstrate that recurrent models and contextual language embeddings are essential for success, and that agents can achieve zero-shot generalization across similar procedural domains.
Introduction
Most text-based games (TBGs) used in AI research, such as Zork or the Jericho benchmark, are set in fictional, high-fantasy worlds. While these environments test linguistic reasoning, the knowledge acquired—such as how to navigate a dungeon—lacks real-world utility. Humans, by contrast, navigate the world using "Script Knowledge": an implicit understanding of the stereotypical sequences of actions required to complete a task.
Existing environments lack grounding in these daily human activities. Furthermore, real-world tasks are non-linear and linguistically diverse; there are many ways to describe the same step and multiple valid paths to reach a goal. ScriptWorld aims to bridge this gap by creating a complex, flexible environment based on actual human descriptions of daily life.
Methods
The construction of ScriptWorld involves a multi-step pipeline designed to capture the nuances of human activity.
Graph Construction
The environment is built using human-written "Event Sequence Descriptions" (ESDs) converted into directed graphs. To handle the inherent complexity of real-life tasks, event nodes are split into "entry" and "exit" points. This structure allows for parallel paths—for instance, choosing between taking the stairs or an elevator to reach a specific floor—reflecting the non-linear nature of real-world procedures.
Dynamic Choice Generation and RL Baselines
At each step, the agent is presented with a list of choices. Correct actions are sampled from the current graph node, while "invalid" actions are sampled from distant nodes to test the agent's understanding of temporal order. We evaluated several RL algorithms, including DQN, A2C, PPO, and a recurrent version (RPPO). To leverage prior linguistic knowledge, we utilized pre-trained Sentence-BERT (SBERT) to convert textual observations and actions into vector embeddings. Additionally, an optional "handicap" system using GPT-2 generated hints was implemented to assist in learning.
Datasets
ScriptWorld primarily draws from the DeScript dataset, which contains 40 daily scenarios with 100 human-written sequences each. The current benchmark features 10 "gold-aligned" scenarios, manually verified for accuracy, including Taking a Bath, Baking a Cake, Flying in an Airplane, and Repairing a Flat Tire.
Results and Key Insights
Our evaluation revealed several critical aspects of procedural learning in text environments.
The Importance of Memory and Context
The Recurrent PPO (RPPO) agent consistently outperformed standard PPO. In the "Shopping" scenario without hints, RPPO achieved a score of 8.79 (out of 10), while the non-recurrent DQN struggled at -7.28. This suggests that procedural tasks require a robust "memory" of previous actions to inform future decisions. Furthermore, agents using contextual embeddings (SBERT) performed significantly better than those using non-contextual embeddings like GloVe, highlighting that word meaning in procedural tasks is highly dependent on the surrounding steps.
Complexity and Generalization
The environment presents a high degree of difficulty; for example, the "Flying in an Airplane" scenario contains approximately $2.6 \times 10^{30}$ possible correct paths. Despite this complexity, we observed evidence of Zero-Shot Generalization. Agents trained on "Riding a Bus" showed positive transfer when tested on "Going on a Train," indicating the learning of abstract procedural concepts (e.g., waiting, boarding, paying) rather than simple string memorization.
Exploration Dynamics
We identified an "Exploration Dip" during training, where performance dropped as the agent reached maximum node coverage in the scenario graph. This indicates a phase transition from exploring the environment to the more difficult task of optimizing specific action sequences.
Conclusion and Future Work
ScriptWorld provides a novel benchmark for grounding RL agents in real-world procedural knowledge. While current agents select from a list of options, future work will explore "parser-based" versions requiring free-form text commands. Additionally, we aim to scale the environment to hundreds of activities and incorporate human-in-the-loop feedback to further refine the agent's common-sense reasoning.
Summary updated