ScriptWorld: Text Based Environment for Learning Procedural Knowledge

Abhinav Joshi, Areeb Ahmad, Umang Pandey, Ashutosh Modi

Abstract

ScriptWorld is a novel text-based reinforcement learning (RL) environment designed to evaluate and train agents on "script knowledge"—the procedural common sense required for everyday tasks. Unlike traditional fantasy-based text games, ScriptWorld leverages human-annotated scripts to create complex, branching scenarios for real-world chores like baking a cake or taking a bus. Our experiments demonstrate that while these tasks present significant challenges for standard RL baselines, the integration of contextual embeddings and recurrent architectures allows agents to capture and generalize procedural logic across related domains.

Introduction

Reinforcement learning in text-based environments has traditionally focused on fictional fantasy worlds, such as those found in the Jericho framework or classic games like Zork. While these environments test logic and memory, the knowledge acquired—such as navigating dungeons or using magical items—rarely translates to real-world utility.

Humans navigate daily life using "Script Knowledge": an implicit understanding of the stereotypical sequences of events involved in common activities. For instance, the task of "washing dishes" involves a series of fine-grained steps including scraping food, applying soap, scrubbing, and rinsing. Current AI agents often lack this grounded, procedural common sense. ScriptWorld addresses this gap by providing a benchmark specifically focused on daily human activities.

Example of varied human-annotated scripts for the 'washing dishes' scenario in ScriptWorld.

The ScriptWorld Environment

The authors developed ScriptWorld as a choice-based interaction model built upon the DeScript corpus, a crowd-sourced collection of Event Sequence Descriptions (ESDs).

Graph Construction

The environment is structured as "scenario graphs" where each node represents a sub-task (e.g., "add soap") and edges represent the logical flow between them. To mirror real-world flexibility, the graphs allow for parallel paths (e.g., choosing between stairs or an elevator) and linguistic variations in task descriptions.

Dynamic Interaction

At each step, an agent is presented with a list of textual choices. These include a correct action that advances the task and several "distractors" sampled from other parts of the script or unrelated actions. This setup requires the agent to not only identify the next logical step but also to distinguish it from contextually similar but temporally incorrect actions.

Methods and Baselines

To establish performance benchmarks, the researchers evaluated several standard RL algorithms, including DQN, A2C, PPO, and Recurrent PPO (RPPO).

A critical component of the agent architecture is the use of pre-trained SBERT (Sentence-BERT) models. These models convert textual descriptions into dense vector embeddings, providing the RL agents with a rich, contextual representation of the state space. Additionally, a "handicap" system using GPT-2 was implemented to provide textual hints, helping to calibrate the difficulty of the environment for learning agents.

Key Results and Insights

The benchmark includes 10 daily activities with vast state spaces; for example, the "Flying in an Airplane" scenario contains over $10^{30}$ possible correct paths.

Algorithm Performance

Recurrent PPO (RPPO) emerged as the most effective baseline. Its success is attributed to its recurrent (LSTM) architecture, which enables the agent to maintain a memory of completed sub-tasks within long, sequential procedures.

Contextual Understanding and Generalization

The study found that contextual embeddings (SBERT) significantly outperform static word vectors like GloVe. In procedural tasks, the meaning of a word is highly dependent on its temporal and task-specific context. Furthermore, agents demonstrated "procedural generalization." An agent trained on the "Ride a Bus" scenario performed significantly better than a random baseline when introduced to a "Go on a Train" scenario, suggesting that the models are learning the underlying logic of public transportation rather than merely memorizing text strings.

Conclusion

ScriptWorld provides a vital bridge between text-based RL and real-world procedural knowledge. While the current benchmark is limited to 10 high-quality scenarios, it highlights the complexity of learning human procedures from scratch. Future work will focus on automating script alignment to expand the environment to hundreds of activities and transitioning from choice-based to parser-based (free-form text) interactions to further challenge agent capabilities.

Summary updated

Abstract

Text-based games provide a framework for developing natural language understanding and commonsense knowledge about the world in reinforcement learning based agents. Existing text-based environments often rely on fictional situations and characters to create a gaming framework and are far from real-world scenarios. In this paper, we introduce ScriptWorld: a text-based environment for teaching agents about real-world daily chores and hence imparting commonsense knowledge. To the best of our knowledge, it is the first interactive text-based gaming framework that consists of daily real-world human activities designed using scripts dataset. We provide gaming environments for 10 daily activities and perform a detailed analysis of the proposed environment. We develop RL-based baseline models/agents to play the games in ScriptWorld. To understand the role of language models in such environments, we leverage features obtained from pre-trained language models in the RL agents. Our experiments show that prior knowledge obtained from a pre-trained language model helps to solve real-world text-based gaming environments.
BibTeX
@inproceedings{joshi2023scriptworld,
  author = {Abhinav Joshi and Areeb Ahmad and Umang Pandey and Ashutosh Modi},
  title = {{ScriptWorld: Text Based Environment for Learning Procedural Knowledge}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/566},
}
Figure