Textual Belief States for World Models: Identifiable Representation Learning Under Strict Mediation

Xiang Gao, Kaiwen Dong, Yuguang Yao, Padmaja Jonnalagedda, Kamalika DasarXiv 2026 · arXiv

TL;DR

Modern LLM-based world models often "cheat" by bypassing their own latent states and reading the full interaction history, making their internal representations uninformative and unidentifiable. This paper introduces a method to enforce strict latent state mediation using discrete, human-readable textual states (JSON predicates) trained via a novel tree-structured reinforcement learning algorithm called Factorized GRPO (fGRPO). This approach dramatically improves representation quality and long-horizon rollout stability without sacrificing one-step prediction accuracy.

What Problem Does This Paper Address?

To plan and reason in partially observed environments, an agent needs a "world model" that maintains an internal latent state summarizing the history of interactions. In classical reinforcement learning, architectures enforce a strict bottleneck: the model must predict the next observation using only its current latent state and the immediate action.

However, modern Large Language Model (LLM) based world models frequently relax this constraint. Because LLMs have massive context windows, decoders often have access to both the latent state and the full observation history. The authors call this the history bypass or leaky bypass problem. When a model can simply look back at the raw history to make a prediction, it can achieve high predictive accuracy while its explicit latent state remains completely uninformative—a challenge known as the identifiability problem.

Enforcing a strict bottleneck in text-based environments is incredibly difficult because textual latent states are discrete and non-differentiable, ruling out standard variational training methods (like those used in continuous world models like Dreamer). Expressive LLM decoders trained on standard objectives simply suffer from "posterior collapse" and ignore the bottleneck entirely.

Figure 1: Three formulations of a world model in a partially observed setting: (a) Stateless, (b) LeakyState, and (c) StrictState. StrictState is proposed to be trained using (d) factorized GRPO.

Methods

The authors propose a framework that combines a strict architectural bottleneck with a text-specific reinforcement learning training recipe:

  1. Strict Latent State Mediation: The architecture completely cuts off the decoder's access to the interaction history once a latent state is generated. Predictions must flow exclusively through the latent state.
  2. Textual Latent States: Instead of continuous vectors, the latent state is represented explicitly as text—specifically, a JSON list of predicate-arguments (e.g., [{"predicate": "at", "arguments": ["P", "kitchen"]}]). This matches the LLM's native text modality, provides an adaptive bottleneck (variable length), and is fully interpretable by humans.
  3. Factorized GRPO (fGRPO): Standard Group Relative Policy Optimization (GRPO) operates on a single autoregressive prompt, which would inherently leak history to the decoder. To fix this, the authors introduce fGRPO, a tree-structured extension. For a given history, it samples a tree of candidate latent states, next states, and observations. Rewards are computed at the leaves (by comparing predicted vs. ground-truth observations) and averaged upward through the tree, allowing separate, group-relative advantage updates for the encoder, transition, and decoder steps.
  4. Reward Design: The reward function penalizes formatting errors and rewards fact-level observation accuracy. Crucially, it includes auxiliary rewards for compactness: deduplication (penalizing redundant facts using a Natural Language Inference model) and ablation (measuring if removing a fact hurts prediction).

Datasets

The approach was evaluated offline on two highly structured, partially observed text-based benchmarks:

  • TextWorld: A framework for text-based games, configured here with varying world sizes (1–12 rooms) and recipe lengths (1–5 ingredients) to test compositional difficulty.
  • ScienceWorld: A complex text environment simulating elementary science tasks.

Both environments feature fully specified underlying symbolic states that are hidden from the agent but used by the researchers for evaluation.

Key Insights and Ideas

  • Predictive Accuracy as a Diagnostic: Under strict mediation, optimal predictive accuracy mathematically guarantees that the latent state is a sufficient statistic of history (Proposition 1). In leaky architectures, excellent accuracy can coexist with a completely useless latent state (Proposition 2). Strict mediation makes representation quality empirically testable.
  • Mitigating Error Propagation: In leaky or stateless models, small errors compound geometrically over time because the model relies on an expanding history of its own imperfect past predictions. Strict mediation forces the error to propagate through a single, fixed-size state bottleneck, leading to bounded error growth under contractive dynamics (Proposition 3).
  • Better Compression Calibration: Strictly mediated models naturally learn to scale their state length based on environmental complexity. Leaky models tend to over-compress because they rely on the history bypass to do the heavy lifting.
Figure 2: Compression ratio. CR = history words

Key Results

  • Maintained Accuracy, Higher Quality: On both TextWorld and ScienceWorld, the strictly mediated model (StrictState-Text) matched the one-step prediction accuracy (obs.F1) of leaky and stateless baselines, but achieved significantly higher state accuracy (state.F1)—up to +30% for a 0.5B model and +21% for a 4B model in highly complex settings.
  • Massive Long-Horizon Gains: The benefits of strict mediation compound over time. In multi-step rollouts on TextWorld (4B model), StrictState-Text outperformed the leaky baseline by 5% at horizon 1, ballooning to a massive 98% improvement at horizon 9.
  • Ablation Insights: Rewarding observation accuracy alone leads to invalid or poorly structured text states. The inclusion of the support reward ($R_{sup}$) was found to be the single most critical factor for recovering long-horizon rollout performance.

Limitations and Future Work

While highly effective, the authors note several limitations:

  • Encoder Burden: Strict mediation places the entire burden of history compression onto a single-shot encoder. If the encoder misses a detail, it is permanently lost, as the decoder can no longer look back at the history to correct it.
  • Domain Specificity of Rewards: The auxiliary rewards rely on structured fact extraction (JSON predicates). Extending this to less structured, open-ended text domains will require more flexible evaluation metrics, such as learned similarity models or LLM-as-a-judge scoring.
  • Downstream Application: The evaluation focused strictly on representation and world-modeling quality; demonstrating how these improved states translate directly into better downstream planning and policy execution remains future work.

Summary updated

Abstract

World models in partially observed environments rely on latent representations that summarize interaction history, but in many modern LLM-based architectures predictive performance fails to reflect representation quality due to history bypass, rendering the latent state unidentifiable. Strict latent state mediation, requiring predictions to depend only on the latent state and action, is a classical principle that resolves this, but enforcing it in text-based settings is an open challenge: textual latent states are discrete and non-differentiable, precluding variational training, and expressive LLM decoders readily ignore the bottleneck. We show how to make strict mediation work in the text domain. We formalize why it is necessary, showing that strict mediation makes representation quality empirically testable while history-leaky architectures break this connection. We then introduce textual latent states, which are discrete, interpretable, and variable-length, and factorized GRPO (fGRPO), a tree-structured reinforcement learning method that enforces strict mediation during training. Experiments on TextWorld and ScienceWorld show preserved one-step prediction accuracy alongside up to 57\% gains in representation quality and 98\% improvements in rollout performance, increasing with task complexity and horizon.
BibTeX
@misc{gao2026textual,
  author = {Xiang Gao and Kaiwen Dong and Yuguang Yao and Padmaja Jonnalagedda and Kamalika Das},
  title = {{Textual Belief States for World Models: Identifiable Representation Learning Under Strict Mediation}},
  year = {2026},
  eprint = {2606.27681},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.27681},
}
Figure