The Challenge of Text-Based Games
Text-based games, such as the classic interactive fiction Zork, present unique challenges for standard Reinforcement Learning (RL) agents. Unlike games like Chess or Atari, where the state is a fully observable grid or a set of pixels, text games are played entirely through natural language. This introduces two massive hurdles for AI:
- Partial Observability: An agent only "sees" what is described in its immediate location. It must remember that a brass lantern was left in the living room even after moving to the cellar.
- Combinatorial Action Space: At any given moment, there are roughly $10^{14}$ possible strings of text an agent could type. While most are nonsensical (e.g., "eat house"), the agent does not inherently know this.
To successfully navigate these environments, agents need a World Model—an internal representation that predicts how the world changes when they act and identifies which actions are actually possible. While previous attempts have used Knowledge Graphs (KGs) to track the state of the world, they have historically struggled to accurately predict how those graphs should evolve over time.
Introducing Worldformer
To address these challenges, researchers have introduced Worldformer, a multi-task transformer architecture designed specifically to build world models for text-based environments. Worldformer frames world modeling as a "Set of Sequences" generation task, bringing three core innovations to the table:
1. Knowledge Graph Difference Generation
Instead of attempting to predict the entire next state of the world ($G_{t+1}$), Worldformer predicts the difference or "delta" ($G_{t+1} - G_t$). It focuses exclusively on what was added to the world (e.g., "you now have the leaflet"). Deletions are handled by a set of logical rules (e.g., if you move to a new room, you are no longer in the old one). By focusing on the delta, the model reduces the average number of tokens it needs to generate per step from approximately 26 down to just 10, making the output space much smaller and easier to learn.
2. Multi-task Learning
The model simultaneously learns to (1) update the Knowledge Graph and (2) predict the set of "valid actions" (actions that will actually change the game state). The authors hypothesize that these tasks are mutually informative: knowing what you can do helps you predict what will change, and vice versa. A shared encoder allows the model to build a richer representation of the environment's underlying "physics."
3. Set of Sequences (SOS) Loss
Standard language models assume a strict order (e.g., word A must follow word B). However, a Knowledge Graph is a set of facts, and valid actions are a set of strings—the order in which they are listed does not matter. Treating a list of actions as a single long string confuses standard Seq2Seq models because there is no inherent "correct" order. To solve this, the authors introduced a Set of Sequences (SOS) loss function that is permutation-invariant for the items in the set, while maintaining sequence order within each individual item.
Datasets and Key Results
The study utilized the JerichoWorld Dataset, which contains over 24,000 mappings between natural language observations, KGs, and valid actions across 27 different text games for training. To test zero-shot generalization, the model was evaluated on 9 "never-before-seen" games.
Worldformer was tested against several baselines, including rule-based extractors and QA-based models like Q*BERT, yielding impressive results:
- Knowledge Graph Prediction: Worldformer achieved a 39.15% Exact Match score on graph-level metrics, significantly outperforming the next best baseline (GATA-W at 24.06%).
- Valid Action Generation: It achieved a 23.22% Exact Match score, outperforming the CALM baseline (13.79%).
Crucially, an ablation study proved that all three components—graph differences, multi-tasking, and SOS loss—were necessary. Removing the "graph difference" simplification caused the largest drop in performance, proving that simplifying the output space is critical for modeling complex textual worlds.
Limitations and Future Work
While Worldformer represents a significant leap forward, the authors note a few remaining hurdles:
- Zero-Shot Difficulty: An Exact Match score of ~39% on new games shows that generalizing to completely different genres (e.g., moving from a fantasy game to a sci-fi game) remains a high bar.
- Downstream RL Integration: This paper focuses strictly on the world model itself. The next logical step is integrating this model into a model-based RL agent to evaluate if it leads to faster game completion and better planning.
- Hallucination: Like many generative models, there is a risk of the model "hallucinating" facts that aren't in the text. However, the structured nature of the Knowledge Graph helps mitigate this compared to pure text generation.
Summary updated