Abstract
World models improve a learning agent's ability to efficiently operate in interactive and situated environments. This work focuses on the task of building world models of text-based game environments. Text-based games, or interactive narratives, are reinforcement learning environments in which agents perceive and interact with the world using textual natural language. These environments contain long, multi-step puzzles or quests woven through a world that is filled with hundreds of characters, locations, and objects. Our world model learns to simultaneously: (1) predict changes in the world caused by an agent's actions when representing the world as a knowledge graph; and (2) generate the set of contextually relevant natural language actions required to operate in the world. We frame this task as a Set of Sequences generation problem by exploiting the inherent structure of knowledge graphs and actions and introduce both a transformer-based multi-task architecture and a loss function to train it. A zero-shot ablation study on never-before-seen textual worlds shows that our methodology significantly outperforms existing textual world modeling techniques as well as the importance of each of our contributions.
Introduction
World models are a cornerstone of efficient reinforcement learning (RL), allowing agents to simulate environment dynamics and plan trajectories internally. While highly successful in visual domains like Atari, world modeling for text-based games—or interactive narratives—presents unique challenges. These environments are partially observable, providing only textual descriptions of the immediate surroundings, and feature a combinatorially massive action space (often exceeding $10^{14}$ possible commands).
To navigate these complexities, agents require structured memory to track the state of the world. Knowledge Graphs (KGs) have emerged as a powerful representation for this purpose, but previous research has largely focused on either extracting KGs from observations or predicting valid actions in isolation. This paper introduces Worldformer, a unified transformer-based world model designed to simultaneously predict how an environment's KG will change following an action and generate the set of contextually valid actions available to the agent.
Methodology
The Worldformer architecture introduces three key innovations to make world modeling in textual environments tractable and accurate.
Graph Difference Prediction
A significant hurdle in KG-based world modeling is the complexity of generating entire graph states. Worldformer simplifies this by predicting the graph difference ($\Delta G = G_{t+1} - G_t$) rather than the full next state. By focusing only on the additions and deletions of triples (e.g., an object moving from a room to the player's inventory), the model reduces the average prediction sequence length from approximately 26 tokens to just 10, making the learning task significantly more efficient.
Multi-task Architecture
The model utilizes a multi-task transformer framework. A BERT-like encoder processes the current textual observation, the current KG state, and the proposed action. This shared representation is then fed into two parallel GPT-2-like decoders. One decoder predicts the graph differences, while the other generates the set of valid actions for the next state. This joint training allows the model to leverage the deep interdependencies between world dynamics and action availability.
Set of Sequences (SOS) Loss
Knowledge graphs (sets of triples) and valid actions (sets of phrases) are inherently permutation-invariant—the order in which they are listed does not change their meaning. Standard sequence-to-sequence (Seq2Seq) models, however, penalize models for generating items in a different order than the training data. To address this, the authors developed a custom Set of Sequences (SOS) loss function. This allows the autoregressive decoders to account for the set-based structure of the output, leading to more robust generalization.
Experimental Evaluation
Datasets
The model was trained and evaluated using the JerichoWorld dataset, which contains 24,198 instances of natural language observations, ground-truth KGs, and valid actions. The data was split into 27 training games and 9 held-out test games to evaluate zero-shot generalization to entirely unseen textual worlds. Additionally, the CALM dataset (human gameplay transcripts) was used as a baseline for action generation.
Results
In zero-shot evaluations on unseen games, Worldformer significantly outperformed existing baselines:
- Knowledge Graph Generation: Worldformer achieved a Graph-level F1 score of 41.06%, nearly doubling the performance of the next best baseline, GATA-W (25.19%).
- Valid Action Generation: The model achieved an F1 score of 25.54%, surpassing both standard Seq2Seq (19.44%) and CALM-based (19.11%) models.
Ablation studies confirmed that the graph difference simplification was the primary driver of success, while the SOS loss and multi-task training were essential for achieving state-of-the-art results.
Conclusion
Worldformer demonstrates that framing world modeling as a set-of-sequences generation problem over graph differences allows for effective zero-shot generalization in complex textual environments. While the model currently reflects the fictional and sometimes non-normative scenarios found in text games, it provides a powerful foundation for future model-based RL agents. Integrating Worldformer into an active agent is the next step toward achieving human-level sample efficiency in interactive narratives.
Summary updated