TL;DR
Modern LLM-based world models often "cheat" by bypassing their own latent states and reading the full interaction history, making their internal representations uninformative and unidentifiable. This paper introduces a method to enforce strict latent state mediation using discrete, human-readable textual states (JSON predicates) trained via a novel tree-structured reinforcement learning algorithm called Factorized GRPO (fGRPO). This approach dramatically improves representation quality and long-horizon rollout stability without sacrificing one-step prediction accuracy.
What Problem Does This Paper Address?
To plan and reason in partially observed environments, an agent needs a "world model" that maintains an internal latent state summarizing the history of interactions. In classical reinforcement learning, architectures enforce a strict bottleneck: the model must predict the next observation using only its current latent state and the immediate action.
However, modern Large Language Model (LLM) based world models frequently relax this constraint. Because LLMs have massive context windows, decoders often have access to both the latent state and the full observation history. The authors call this the history bypass or leaky bypass problem. When a model can simply look back at the raw history to make a prediction, it can achieve high predictive accuracy while its explicit latent state remains completely uninformative—a challenge known as the identifiability problem.
Enforcing a strict bottleneck in text-based environments is incredibly difficult because textual latent states are discrete and non-differentiable, ruling out standard variational training methods (like those used in continuous world models like Dreamer). Expressive LLM decoders trained on standard objectives simply suffer from "posterior collapse" and ignore the bottleneck entirely.
Methods
The authors propose a framework that combines a strict architectural bottleneck with a text-specific reinforcement learning training recipe:
- Strict Latent State Mediation: The architecture completely cuts off the decoder's access to the interaction history once a latent state is generated. Predictions must flow exclusively through the latent state.
- Textual Latent States: Instead of continuous vectors, the latent state is represented explicitly as text—specifically, a JSON list of predicate-arguments (e.g.,
[{"predicate": "at", "arguments": ["P", "kitchen"]}]). This matches the LLM's native text modality, provides an adaptive bottleneck (variable length), and is fully interpretable by humans. - Factorized GRPO (fGRPO): Standard Group Relative Policy Optimization (GRPO) operates on a single autoregressive prompt, which would inherently leak history to the decoder. To fix this, the authors introduce fGRPO, a tree-structured extension. For a given history, it samples a tree of candidate latent states, next states, and observations. Rewards are computed at the leaves (by comparing predicted vs. ground-truth observations) and averaged upward through the tree, allowing separate, group-relative advantage updates for the encoder, transition, and decoder steps.
- Reward Design: The reward function penalizes formatting errors and rewards fact-level observation accuracy. Crucially, it includes auxiliary rewards for compactness: deduplication (penalizing redundant facts using a Natural Language Inference model) and ablation (measuring if removing a fact hurts prediction).
Datasets
The approach was evaluated offline on two highly structured, partially observed text-based benchmarks:
- TextWorld: A framework for text-based games, configured here with varying world sizes (1–12 rooms) and recipe lengths (1–5 ingredients) to test compositional difficulty.
- ScienceWorld: A complex text environment simulating elementary science tasks.
Both environments feature fully specified underlying symbolic states that are hidden from the agent but used by the researchers for evaluation.
Key Insights and Ideas
- Predictive Accuracy as a Diagnostic: Under strict mediation, optimal predictive accuracy mathematically guarantees that the latent state is a sufficient statistic of history (Proposition 1). In leaky architectures, excellent accuracy can coexist with a completely useless latent state (Proposition 2). Strict mediation makes representation quality empirically testable.
- Mitigating Error Propagation: In leaky or stateless models, small errors compound geometrically over time because the model relies on an expanding history of its own imperfect past predictions. Strict mediation forces the error to propagate through a single, fixed-size state bottleneck, leading to bounded error growth under contractive dynamics (Proposition 3).
- Better Compression Calibration: Strictly mediated models naturally learn to scale their state length based on environmental complexity. Leaky models tend to over-compress because they rely on the history bypass to do the heavy lifting.
Key Results
- Maintained Accuracy, Higher Quality: On both TextWorld and ScienceWorld, the strictly mediated model (
StrictState-Text) matched the one-step prediction accuracy (obs.F1) of leaky and stateless baselines, but achieved significantly higher state accuracy (state.F1)—up to +30% for a 0.5B model and +21% for a 4B model in highly complex settings. - Massive Long-Horizon Gains: The benefits of strict mediation compound over time. In multi-step rollouts on TextWorld (4B model),
StrictState-Textoutperformed the leaky baseline by 5% at horizon 1, ballooning to a massive 98% improvement at horizon 9. - Ablation Insights: Rewarding observation accuracy alone leads to invalid or poorly structured text states. The inclusion of the support reward ($R_{sup}$) was found to be the single most critical factor for recovering long-horizon rollout performance.
Limitations and Future Work
While highly effective, the authors note several limitations:
- Encoder Burden: Strict mediation places the entire burden of history compression onto a single-shot encoder. If the encoder misses a detail, it is permanently lost, as the decoder can no longer look back at the history to correct it.
- Domain Specificity of Rewards: The auxiliary rewards rely on structured fact extraction (JSON predicates). Extending this to less structured, open-ended text domains will require more flexible evaluation metrics, such as learned similarity models or LLM-as-a-judge scoring.
- Downstream Application: The evaluation focused strictly on representation and world-modeling quality; demonstrating how these improved states translate directly into better downstream planning and policy execution remains future work.
Summary updated