Beyond State Consistency: Behavior Consistency in Text-Based World Models

Youling Huang, Guanqiao Chen, Junchi Yao, Lu Wang, Fangkai Yang, Chao Du, ChenZhuo Zhao, Pu Zhao, Qingwei Lin, Saravan Rajmohan, Dongmei ZhangarXiv 2026 · arXiv

TL;DR

Current text-based world models are trained to mimic environment text word-for-word, but this often fails to preserve how an agent actually behaves inside the simulation. This paper introduces Behavior Consistency Training, a reinforcement learning paradigm that optimizes world models using a novel Behavior Consistency Reward (BehR). By aligning the action probabilities of a reference agent between the real and simulated states, the authors create simulators that are significantly more reliable for offline evaluation and lookahead planning.

What Problem Does This Paper Address?

When developing interactive AI agents for tasks like web navigation or text games, testing them in real-world environments can be slow, expensive, or risky. To solve this, researchers build text-based "world models" (WMs)—simulators that predict the next environmental state given the agent's current history and action.

Traditionally, these world models are trained using Supervised Fine-Tuning (SFT) to maximize token-level similarity (e.g., Exact Match or BERTScore) between the predicted text and the ground-truth text. However, the authors identify a critical flaw in this approach: metric inversion.

Figure 1: Metric inversion in WebShop. The Bad State drops the target product and changes the agent’s action despite high surface similarity, while the Good State drops irrelevant products yet preserves the real action. BehR scores this behavioral effect rather than surface overlap.

As shown in Figure 1, a simulator might generate a web page that matches 97% of the original text but accidentally omits the specific item the agent needs to buy. Despite a high similarity score, the agent fails. Conversely, a simulator could hallucinate or omit irrelevant text, scoring poorly on surface metrics, yet still preserve the exact cues the agent needs to succeed. This paper shifts the goal from state consistency (making the text look identical) to functional consistency (ensuring the text induces the same agent behavior).

Methods

To achieve functional consistency, the authors introduce a reinforcement learning framework using Group Relative Policy Optimization (GRPO) driven by a new step-level reward function: the Behavior Consistency Reward (BehR).

State consistency vs. behavior consistency training pipelines.

Instead of comparing the generated text directly to the ground truth, BehR utilizes a frozen, external LLM as a Reference Agent ($\pi_{ref}$). Here is how the reward loop works:

  1. The world model generates $n$ candidate next-state text descriptions.
  2. The Reference Agent calculates the log-probability of the logged next expert action under both the real environment state ($s_t$) and the world model's predicted state ($\hat{s}_t$).
  3. The reward is formulated as the negative absolute difference between these two log-probabilities: $$R_{beh} = -|\bar{\ell}_{pred} - \bar{\ell}_{real}|$$
  4. GRPO optimizes the world model to minimize this gap, effectively training it to preserve the behavioral difficulty and decision pathways of the real environment.

To evaluate task-level success, the authors emphasize the Pairwise Consistency Ratio ($CR_{pw}$), which measures the fraction of successful real-world tasks that remain successful when the exact action sequence generated inside the world model is replayed back in the real environment.

Datasets

The methods were validated across two distinct text-based interactive environments:

  • WebShop: An online shopping environment consisting of open catalog search pages and item detail pages. It represents a highly open-ended domain where the world model must generate coherent, internally consistent product listings.
  • TextWorld: A procedurally generated text adventure game framework governed by strict logic rules (e.g., tracking inventory and room connectivity across multi-step dependency chains).

Training data was derived by restructuring step-level tuples from the existing Word2World supervised corpus, yielding 4,321 WebShop training tuples and 6,000 TextWorld training tuples after difficulty-aware filtering.

Key Insights and Ideas

  • Surface Metrics are Blind to Behavior: In controlled perturbation tests (Table 1), standard metrics like BERTScore and F1 rated severe, task-breaking errors (like deleting the target object) as less severe than minor changes (like deleting irrelevant text). BehR was the only metric that ranked the severity of these changes correctly.
  • Symmetry Prevents "Too Easy" Simulators: Because BehR uses an absolute difference, it penalizes the world model not only for making tasks harder but also for making them artificially easier (e.g., omitting constraints or over-simplifying text), which is vital for accurate simulator calibration.
  • Decoupled from the Target Agent: The world model does not need to be trained against the exact agent it will evaluate. The authors demonstrated that a world model optimized using a Qwen3-8B reference agent successfully transferred its behavioral alignment benefits to entirely different evaluation agents, including GPT-4o and GPT-5.

Key Results

The authors evaluated their behavior-consistent world models (BehR-WM) against the standard supervised baseline (W2W-WM) and a token-level reinforcement learning control (F1-WM).

Table 3: Task-level functional consistency. Real/WM/W2R are success rates in the real environment, simulated environment, and real-environment replay of WM actions. CR = W2R/Real; CRpw is the fraction of Real-successful tasks preserved under W2R and is our primary metric. Bold marks the best CRpw value within each agent/backbone/- domain group. F1-WM is a matched GRPO control; the additional SurR-WM control is reported in Appendix D.
  • Substantial Task-Level Gains: As detailed in Table 3 (Figure 6), BehR-WM consistently achieved the highest $CR_{pw}$ across almost all evaluation agent and backbone configurations. For instance, on the Qwen2.5-7B base model in WebShop, BehR raised the $CR_{pw}$ of Qwen3-8B from 0.345 to 0.483, and GPT-4o from 0.760 to 0.840.
  • Preserved Local Accuracy: Optimizing for behavior did not degrade single-step textual generation quality. In fact, on TextWorld, the single-step Exact Match accuracy improved significantly (e.g., from 64.72% to 73.11% for the Qwen2.5 backbone).
  • Reduced False Positives: In surrogate evaluations, standard world models often suffer from a "too easy simulator" effect, causing weak agents to look artificially competent. BehR-WM dramatically reduced these false positives (dropping Qwen3-0.6B's false positive rate on TextWorld from 42.5% to just 9.5%).
  • Better Lookahead Planning: When used as a simulator for inference-time lookahead planning ($K=5$), BehR-WM provided more reliable rollouts, outperforming the baseline SFT world model.

Limitations and Future Work

While promising, the authors note several limitations:

  • Proxy Constraints: BehR evaluates the likelihood of only the logged next action rather than mapping the agent's entire shifting action distribution. This proxy can be less effective in highly complex scenarios where multiple equally valid paths exist.
  • Reference Dependency: Although the model exhibits cross-agent transfer, the performance gains are still partially bounded by the capability and style of the chosen frozen Reference Agent.
  • Evaluation Scale: Each GRPO configuration was trained only once due to compute constraints, leaving run-to-run variance across independent random seeds unquantified.

Future extensions include exploring broader reward families that don't rely on a single reference agent and testing the paradigm against wider decoding variations.

Summary updated

Abstract

World models have been emerging as critical components for assessing the consequences of actions generated by interactive agents in online planning and offline evaluation. In text-based environments, world models are typically evaluated and trained with single-step metrics such as Exact Match, aiming to improve the similarity between predicted and real-world states, but such metrics have been shown to be insufficient for capturing actual agent behavior. To address this issue, we introduce a new behavior-aligned training paradigm aimed at improving the functional consistency between the world model and the real environment. This paradigm focuses on optimizing a tractable step-level metric named Behavior Consistency Reward (BehR), which measures how much the likelihood of a logged next action changes between the real state and the world-model-predicted state under a frozen Reference Agent. Experiments on WebShop and TextWorld show that BehR-based training improves long-term alignment in several settings, with the clearest gains in WebShop and less movement in near-ceiling regimes, while preserving or improving single-step prediction quality in three of four settings. World models trained with BehR also achieve lower false positives in offline surrogate evaluation and show modest but encouraging gains in inference-time lookahead planning.
BibTeX
@misc{huang2026beyond,
  author = {Youling Huang and Guanqiao Chen and Junchi Yao and Lu Wang and Fangkai Yang and Chao Du and ChenZhuo Zhao and Pu Zhao and Qingwei Lin and Saravan Rajmohan and Dongmei Zhang},
  title = {{Beyond State Consistency: Behavior Consistency in Text-Based World Models}},
  year = {2026},
  eprint = {2604.13824},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.13824},
}
Figure