MemWM: Memory-Augmented Text-Based World Model

Yujun Wang, Tao Zhang, Jinhe Bi, Aniri, Wenxuan Ye, Boliang Liu, Sikuan Yan, Shuning Wang, Xuebing Zhou, Sören Pirk, Hinrich Schütze, Yunpu MaarXiv 2026 · arXiv

TL;DR

Text-based world models used by language agents often suffer from a "state-fidelity bottleneck," generating fluent but factually flawed next-state predictions that ruin lookahead planning. This paper introduces MemWM, a framework that equips world models with a curated memory bank of transition rules and state caches to ensure highly accurate state imagination. Combined with a novel evaluation metric called Structured State Fidelity (SSF), MemWM significantly boosts both state prediction accuracy and downstream task success across complex interactive environments without needing to train the acting policy.

What Problem Does This Paper Address?

Large language model (LLM) agents increasingly rely on internal "world models" to simulate and predict how an environment will change in response to their actions. This capability allows them to imagine future trajectories and plan before executing a move. However, text-based world models frequently fail to maintain factual accuracy: they might generate highly plausible text that nonetheless forgets where an object was placed, flips a device state, or corrupts a product price.

These subtle errors compound rapidly during lookahead planning, leading the agent astray. Making matters worse, standard text evaluation metrics fail to diagnose this bottleneck. Exact Match is too strict, penalizing harmless paraphrasing, while Word F1 is too permissive, giving high scores to sentences that share most tokens but swap critical, behavior-determining details (e.g., predicting "the fridge is open" instead of "closed").

Figure 1: Illustration of the state-fidelity bottleneck in world modeling. Upper: Given the current state st and action at, a world model without retrieved memory may produce a fluent but low-fidelity next state, hallucinating objects, missing task-relevant facts, or applying the wrong transition rule. Lower: By retrieving curated world memory, MEMWM conditions prediction on relevant state caches, transition rules, and hard-to-predict facts, producing an imagined state that better matches the ground truth st+1.

Methods

The authors propose MemWM (Memory-Augmented Text-Based World Model), which introduces a dual-channel memory architecture to assist lookahead planning:

  1. World Memory (Model-side): A curated bank storing transition rules, state caches, and hard-to-predict facts distilled from past experiences. At inference time, relevant entries are retrieved based on the current context and candidate action, then injected directly into the world model's prompt as auxiliary evidence to guide next-state imagination.
  2. World Skill (Policy-side): A separate channel providing task-level skills (high-level priors at the start of an episode) and step-wise corrective guidance (dynamic local reminders) to help a frozen policy model propose better candidate actions.

To optimize the world model, the authors employ Supervised Fine-Tuning (SFT) followed by reinforcement learning via Group Relative Policy Optimization (GRPO), utilizing memory-augmented training data.

Additionally, they introduce Structured State Fidelity (SSF), a domain-aware metric that projects both predicted and ground-truth states into structured facts or fields before comparing them component-wise. This ensures evaluation tracks actual behavior-relevant information rather than surface string overlap.

Figure 2: Overview of the MEMWM planning pipeline. Top: The left branch builds reusable state and transition memories; the right branch builds reusable action-selection experience. Bottom: At inference time, the agent retrieves relevant entries, imagines candidate next states, scores the candidates, and commits to the highest-scoring action in the environment.

Datasets

The framework and metric are evaluated across three distinct text-based decision-making benchmarks:

  • ALFWorld: An embodied household environment requiring multi-room navigation, object manipulation, and state tracking.
  • ScienceWorld: A complex simulator for elementary-school science tasks involving text-based interactions with laboratory devices, substances, and thermodynamic properties.
  • WebShop: A web-navigation and e-commerce environment focusing on product attributes, option selection, and user constraints.
  • CookingWorld (TextWorldExpress): Used as a held-out domain to validate the cross-domain transferability of the SSF metric.

Key Insights and Ideas

  • Surface Metrics Mask World Model Flaws: A world model can achieve a deceptive 0.96 Word F1 score while completely failing to track critical state changes. SSF exposes these hidden failures by focusing strictly on structured facts.
  • Memory as a Constraint, Not Just Context: Externalizing world knowledge into a structured memory bank acts as a set of guardrails during generation, preventing the LLM from violating environment rules it technically "knows" but fails to apply consistently.
  • Empowering Frozen Policies: You don't necessarily need to fine-tune or update the weights of the acting policy agent if your world model can accurately predict the future and provide corrective guidance. A high-fidelity world model naturally unlocks better planning for existing policies.

Key Results

  • Fidelity Boost: Memory-augmented RL training (Mem-Aug RL) achieved the highest SSF scores across all tested backbones (Llama-3.2-1B, Qwen3-4B, Qwen2.5-7B). On WebShop, it improved state fidelity by up to 206.3% compared to standard SFT.
  • Downstream Success: Without modifying or training the policy backbone, MemWM improved downstream task success across the board, yielding up to a 65.4% relative gain over an SFT-trained world model agent.
  • Active Memory Reliance: Sensitivity analyses showed that dropping out memory blocks at inference time directly degrades the SSF score, proving that the world model actively relies on the retrieved facts rather than treating them as inert context.
Figure 4: SSF sensitivity to world-memory dropout. With Qwen3-4B, increasing prediction-time memory dropout reduces SSF across ALFWorld, WebShop, and ScienceWorld.

Limitations and Future Work

  • Domain-Specific Engineering: Computing the SSF metric and structuring the world memory currently relies on domain-specific rule-based parsers or tiny, specialized LLM extractors. Adapting MemWM to an entirely new environment requires upfront engineering to define schemas and extraction rules.
  • Scalability of Memory Curation: While the paper outlines automated ways to bucket hard negatives and distill trajectories, scaling this curation to open-ended, infinitely complex worlds without manual intervention remains an open challenge.

Summary updated

Abstract

World models are increasingly used to support planning in agents by predicting how environment states evolve in response to agent actions. Yet fluent next-state predictions can still omit task-critical facts, corrupt product attributes, or apply incorrect transition rules. To address such systematic prediction errors, we introduce MemWM, a memory-augmented text-based world model. MemWM uses world memory, a curated memory bank of transition rules, state caches, and hard-to-predict facts, to condition next-state imagination. We evaluate factual state preservation with Structured State Fidelity (SSF), which scores predicted states through benchmark-specific facts and fields. Compared with SFT, memory-augmented training improves SSF by up to 206.3%. In the full planning setting, we keep the policy model frozen and provide policy-side world skill: retrieved task-level skills and step-wise corrective guidance for action selection. Across ALFWorld, WebShop, and ScienceWorld, memory-augmented agents improve downstream success over an SFT-trained world-model agent, with up to a 65.4% relative gain. Sensitivity analyses further show that retrieved memory improves task success and efficiency under different memory and action-budget settings.
BibTeX
@misc{wang2026memwm,
  author = {Yujun Wang and Tao Zhang and Jinhe Bi and Aniri and Wenxuan Ye and Boliang Liu and Sikuan Yan and Shuning Wang and Xuebing Zhou and Sören Pirk and Hinrich Schütze and Yunpu Ma},
  title = {{MemWM: Memory-Augmented Text-Based World Model}},
  year = {2026},
  eprint = {2608.07107},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.07107},
}
Figure