TL;DR
Text-based world models used by language agents often suffer from a "state-fidelity bottleneck," generating fluent but factually flawed next-state predictions that ruin lookahead planning. This paper introduces MemWM, a framework that equips world models with a curated memory bank of transition rules and state caches to ensure highly accurate state imagination. Combined with a novel evaluation metric called Structured State Fidelity (SSF), MemWM significantly boosts both state prediction accuracy and downstream task success across complex interactive environments without needing to train the acting policy.
What Problem Does This Paper Address?
Large language model (LLM) agents increasingly rely on internal "world models" to simulate and predict how an environment will change in response to their actions. This capability allows them to imagine future trajectories and plan before executing a move. However, text-based world models frequently fail to maintain factual accuracy: they might generate highly plausible text that nonetheless forgets where an object was placed, flips a device state, or corrupts a product price.
These subtle errors compound rapidly during lookahead planning, leading the agent astray. Making matters worse, standard text evaluation metrics fail to diagnose this bottleneck. Exact Match is too strict, penalizing harmless paraphrasing, while Word F1 is too permissive, giving high scores to sentences that share most tokens but swap critical, behavior-determining details (e.g., predicting "the fridge is open" instead of "closed").
Methods
The authors propose MemWM (Memory-Augmented Text-Based World Model), which introduces a dual-channel memory architecture to assist lookahead planning:
- World Memory (Model-side): A curated bank storing transition rules, state caches, and hard-to-predict facts distilled from past experiences. At inference time, relevant entries are retrieved based on the current context and candidate action, then injected directly into the world model's prompt as auxiliary evidence to guide next-state imagination.
- World Skill (Policy-side): A separate channel providing task-level skills (high-level priors at the start of an episode) and step-wise corrective guidance (dynamic local reminders) to help a frozen policy model propose better candidate actions.
To optimize the world model, the authors employ Supervised Fine-Tuning (SFT) followed by reinforcement learning via Group Relative Policy Optimization (GRPO), utilizing memory-augmented training data.
Additionally, they introduce Structured State Fidelity (SSF), a domain-aware metric that projects both predicted and ground-truth states into structured facts or fields before comparing them component-wise. This ensures evaluation tracks actual behavior-relevant information rather than surface string overlap.
Datasets
The framework and metric are evaluated across three distinct text-based decision-making benchmarks:
- ALFWorld: An embodied household environment requiring multi-room navigation, object manipulation, and state tracking.
- ScienceWorld: A complex simulator for elementary-school science tasks involving text-based interactions with laboratory devices, substances, and thermodynamic properties.
- WebShop: A web-navigation and e-commerce environment focusing on product attributes, option selection, and user constraints.
- CookingWorld (TextWorldExpress): Used as a held-out domain to validate the cross-domain transferability of the SSF metric.
Key Insights and Ideas
- Surface Metrics Mask World Model Flaws: A world model can achieve a deceptive 0.96 Word F1 score while completely failing to track critical state changes. SSF exposes these hidden failures by focusing strictly on structured facts.
- Memory as a Constraint, Not Just Context: Externalizing world knowledge into a structured memory bank acts as a set of guardrails during generation, preventing the LLM from violating environment rules it technically "knows" but fails to apply consistently.
- Empowering Frozen Policies: You don't necessarily need to fine-tune or update the weights of the acting policy agent if your world model can accurately predict the future and provide corrective guidance. A high-fidelity world model naturally unlocks better planning for existing policies.
Key Results
- Fidelity Boost: Memory-augmented RL training (Mem-Aug RL) achieved the highest SSF scores across all tested backbones (Llama-3.2-1B, Qwen3-4B, Qwen2.5-7B). On WebShop, it improved state fidelity by up to 206.3% compared to standard SFT.
- Downstream Success: Without modifying or training the policy backbone, MemWM improved downstream task success across the board, yielding up to a 65.4% relative gain over an SFT-trained world model agent.
- Active Memory Reliance: Sensitivity analyses showed that dropping out memory blocks at inference time directly degrades the SSF score, proving that the world model actively relies on the retrieved facts rather than treating them as inert context.
Limitations and Future Work
- Domain-Specific Engineering: Computing the SSF metric and structuring the world memory currently relies on domain-specific rule-based parsers or tiny, specialized LLM extractors. Adapting MemWM to an entirely new environment requires upfront engineering to define schemas and extraction rules.
- Scalability of Memory Curation: While the paper outlines automated ways to bucket hard negatives and distill trajectories, scaling this curation to open-ended, infinitely complex worlds without manual intervention remains an open challenge.
Summary updated