A growing number of agents never meet the real world while training. Instead, a language model plays the environment: given the current web page or household state and the agent's action, it writes out what happens next and whether the agent earned a reward. This makes training cheap. It also means the gap between simulation and reality now has a specific cause. The simulator is a language model, and language models hallucinate, meaning they state false things fluently. Whenever the simulated environment invents a product that isn't in the catalog, lets a checkout succeed that would really fail, or leaves out an error message, the agent learns from a world that does not exist.
This proposal treats each simulated step as a set of factual claims about the real environment. It checks a small, targeted sample of those claims against the real environment and spends a fixed budget of real interactions where the simulator is most likely to be wrong. We also test a sharper hypothesis: LLM simulators make mostly optimistic errors, predicting success where reality returns failure. If so, the agent trained on them should inherit that optimism, in the form of the over-eager, unverified actions documented in tool-use and web-search agents. If the hypothesis holds, a single fix aimed at simulator hallucination should reduce both the sim-to-real performance drop and the agent's own tendency to hallucinate.
Core Claim
The sim-to-real gap of an LLM-simulated environment is largely a measurable quantity: the rate of false claims in simulated transitions, compared against the real environment. Most of this gap can be closed by finding and correcting those claims, without adding more real data uniformly. The second claim is causal. Simulator errors that favor success transfer into agent behavior that skips verification and abstains too rarely. Correcting those specific errors should therefore improve how often the agent checks results and declines to answer, beyond what equal amounts of uniformly sampled real data achieve.
This builds on three pieces of prior work, each in a different role. DREAMGYM supplies the setting. It trains agents inside an LLM "experience model" that predicts abstract textual next states and rewards, and it guards against invented transitions by retrieving similar steps from a buffer seeded with 2k–10k real trajectories. That guard is applied uniformly. Every synthetic transition is trusted once retrieval has conditioned it, and nobody measures which transitions remain false. ARTIS motivates the work. It found that standard LLM simulators miss rare failure consequences and fine-tuned a risk-aware simulator on failure-rebalanced data, but it never checks individual simulated tool returns at test time. A Unified Definition of Hallucination supplies the evaluation method. It defines a hallucination as an output whose extracted atomic claims are false relative to an explicit reference world model. The paper is purely conceptual, and it leaves open both claim extraction and worlds that change as the agent acts. In our setting the reference world is concrete: the real WebShop, ALFWorld, or WebArena-Lite instance, replayed to the same state. Finally, UGAT supplies the gating idea. It rejects learned dynamics corrections in traffic control when an evidential uncertainty estimate is high. It works only on numeric state-action dynamics, though, and it falls back to the uncorrected action rather than querying reality.
Research Program
The first stage is measurement. We collect states from the real environments by replaying logged trajectories. At each state the experience model predicts the next state and reward, and the real environment executes the same action. An extractor model breaks both outputs into atomic claims about one transition. Examples are "item X is now in the cart," "the search returned a product priced under $30," "the drawer is open," and "the task is complete." Each simulated claim is then labeled as supported, contradicted, or missing relative to the real outcome. Because ALFWorld and WebShop states are symbolic, most labels can be checked by matching against the environment state. A human audit of a sample measures extractor error, which matters because [ClawTrack](/work/clawtrack-towards-trace-level-evaluation-and-improvement) shows LLM judges are useful but imperfect (expert–LLM agreement $r=0.81$). This stage produces the first claim-level hallucination profile of an LLM world simulator. It also tests the optimism hypothesis directly, by comparing the rate of invented successes with the rate of invented failures.
The second stage is targeted correction. A lightweight detector scores each simulated transition for its probability of containing a contradicted claim. The detector is trained on stage-one labels drawn only from the seed budget. During agent training, the transitions with the highest scores are sent to the real environment, within a fixed budget $B$ of real steps. The real outcome replaces the imagined one in the agent's training data and is added to the retrieval buffer. Flagged transitions beyond the budget are dropped. The budget $B$ is set to match the real data DREAMGYM's sim-to-real warm-start already consumes. Our method therefore uses no extra real interaction; it only chooses where that interaction goes.
The third stage traces how simulator errors carry into the agent's behavior. We evaluate trained agents on the success rate in the real environment. We also measure behaviors that reflect hallucination. Result verification is measured with ClawTrack's process rubric dimension, which that paper identifies as the weakest and most independent dimension. Abstention is measured by whether the agent declines to act on perturbed tasks that cannot be completed. These are built in the style of ICT, whose two perturbations are removing required arguments and swapping in similar but incompatible APIs.
Baselines
The naive reference point is DREAMGYM-S2R as published. The agent trains in the retrieval-guarded experience model and is then fine-tuned on real data whose total size equals our budget $B$. This is the strongest existing method for the question. It is the only one that already trains web and embodied agents in an LLM simulator with an explicit defense against invented transitions, and it reports gains of more than 30% over all baselines on WebArena. No reported absolute WebArena-Lite or ALFWorld success rate was found on Lacuna for DREAMGYM-S2R, so we will rerun it ourselves. We will use its released backbones (Llama-3.1-8B, Qwen-2.5-7B), the same seed buffer, the same number of synthetic rollouts, and the same tuning budget for all arms. Beating it means higher real success at equal real-step count, averaged over three seeds, with non-overlapping confidence intervals on at least two of the three environments.
The key ablation spends the same budget $B$ on real transitions chosen uniformly at random among synthetic steps, instead of choosing them by detector score. This isolates the ingredient we are adding: allocation targeted by hallucination. If random allocation matches targeted allocation, the claim fails.
A risk-aware simulator baseline follows ARTIS and fine-tunes the experience model on failure-rebalanced data drawn from the same seed set. For function-calling, we also report ARTIS's own setting. Its Llama3.1-8B-Instruct agent with the risk-aware simulator reaches 29.5% accuracy on BFCL-v3, and its simulator reaches 95.4% high-fidelity. That is the reference number for our BFCL-v3 extension. Beating it means our gated simulator reaches higher agent accuracy under the same simulated-rollout count. A claim-level audit should also show that ARTIS's 95.4% fidelity hides contradicted claims concentrated in failure cases.
Why not just…?
Why not just add more real seed data to the retrieval buffer? This is the most direct threat, and the random-allocation ablation already tests a strong version of it, since uniform real data is exactly what extra seeds provide. We also run a curve that doubles and quadruples seed size. If uniform seeds close the gap as fast per real step as targeted queries do, we will report that the simulator's errors are spread evenly rather than concentrated. In that case the measurement stage stands as the contribution, and the correction method does not.
Why not just use ensemble disagreement or self-consistency as the uncertainty signal? It needs no labels and might work. Agrawal et al. show that consistency across indirect queries collapses when a model is fabricating, which suggests disagreement carries signal. We therefore include disagreement among five simulator samples as one detector arm. Its weakness is likely to be the optimism hypothesis itself: a simulator that systematically predicts success will agree with itself. Stage one will show whether contradicted claims come with low disagreement. If disagreement matches the trained detector, we adopt it as the cheaper choice.
Why not just fine-tune the simulator on failures, as ARTIS does? That changes the simulator's average behavior, but it does not tell you which transitions are still wrong. Rebalancing may also overcorrect into pessimism, inventing failures that never occur, which would teach excessive refusal. The ARTIS-style baseline, together with our directional error counts, will show whether rebalancing alone removes optimistic errors. We also run its combination with our gating method.
Why not just add a verification checklist to the agent prompt? Structured Verification from ICT raises Gemma-3-4B F1 from 20.90% to 68.57% on incomplete tool conditions. This is a strong fix at test time, and we include it as a prompting control. It addresses the agent's symptom, not the training signal. The causal test is whether agents trained with gating verify more often without the checklist. Comparing the checklist alone, gating alone, and both together separates the two effects.
Proposed Experiments and Decisive Checks
The measurement study runs on WebShop, ALFWorld, and WebArena-Lite, with the DREAMGYM experience model and roughly 5k real transitions per environment. It reports the contradicted-claim rate broken down by claim type and the ratio of invented success to invented failure. The training comparison runs all baseline arms at a matched real-step budget $B$. It reports real success rate, the number of real steps used, and the share of training transitions that still contain contradicted claims. The BFCL-v3 and ACEBench extension repeats the comparison with ARTIS's simulator. Behavioral transfer is measured on ClawTrack-style process scores over agent traces and on abstention F1 for ICT-style perturbed versions of each environment's tasks.
Three results would decide the proposal. First, targeted allocation must beat random allocation at equal budget; if it does not, correction is not the right lever. Second, the measured optimism ratio must be well above one, and agents trained on simulators corrected for optimism must show higher verification and abstention without prompting; if either fails, the causal link between simulator hallucination and agent hallucination is false. Third, the fraction of contradicted claims remaining in training data should predict the real-success gap across all arms. That would make the unified definition's "reference world" a quantity one can actually measure and act on.
References
- Scaling Agent Learning via Experience Synthesis
- ARTIS: Agentic Risk-Aware Test-Time Scaling via Iterative Simulation
- A Unified Definition of Hallucination: It’s The World Model, Stupid!
- Uncertainty-Aware Grounded Action Transformation Towards Sim-to-Real Transfer for Traffic Signal Control
- ClawTrack: Towards Trace-Level Evaluation and Improvement of Real-World Autonomous Agents
- Can Tool-augmented Large Language Models be Aware of Incomplete Conditions?
- Do Language Models Know When They're Hallucinating References?