Language-model agents can often say correctly what another player believes or will do next, and then choose a move that ignores it. Researchers call this the inference-action gap. Most of the evidence comes from static quizzes, and the fixes so far are prompts. We propose a direct causal test that also works in multi-turn interaction. The agent is split into two parts: a belief module, which writes down in a fixed, checkable format what it thinks its partners know and want, and an action module, which reads that record and picks a move. We then edit the written belief and watch whether the action changes the way the edited belief says it should. If it does not, the agent's stated reasoning is decoration. If it does, the gap must sit upstream, in forming the belief, or downstream, in carrying out the action.
This matters because LLM agents are now used as stand-ins for people in social simulations, and in those uses a plausible rationale is not evidence that the rationale drove the behavior. A synthetic survey respondent can give the right answer with the wrong reasons, as shown in Reason-Mediated Behavioral Models for Auditing LLM Social Simulators. Splitting the agent into modules turns "does the agent act on its reasoning" from a judgment about text into an experiment with interventions.
Core Claim
We make two claims. The first is that the inference-action gap in interactive agents should be measured as how strongly actions causally depend on stated beliefs, and that an explicit belief record makes this measurable by counterfactual editing.
The second claim is new. How much raw context the action module is allowed to see, beyond the belief record, sets a measurable tradeoff:
- Full raw context. Privileged information that the partner lacks can override the belief record. This is the "curse of knowledge" documented in SimpleToM.
- Belief record only. Wrong beliefs pass straight into wrong actions. This is the collapse to 42% that How FaR Are Large Language Models From Agents with Theory-of-Mind? reports when its intermediate foresight step is noisy.
We expect the best setting to be intermediate and to depend on the task. We will map this tradeoff rather than assume either extreme.
What Prior Work Establishes and Leaves Open
Four results motivate the proposal by showing that the gap exists.
- SimpleToM finds frontier models above 95% on explicit belief questions but at 49.5% (GPT-4o) on behavior prediction.
- How FaR Are Large Language Models From Agents with Theory-of-Mind? shows GPT-4 falling from 93% on ToMi belief questions to 50% when asked whom to help.
- Position: Theory of Mind Benchmarks are Broken for Large Language Models measures regret, meaning the payoff lost relative to the best response, in repeated games. LLMs with roughly 95% prediction accuracy still lose to a simple tabular reinforcement-learning learner (RMax).
- EnactToM reports 0.0% Pass^3 on hard embodied coordination despite 63.3% on literal belief probes for Gemini-Pro.
Three results constrain the design, because they show that simply surfacing the belief often fixes things.
- Can Large Language Models Serve as Rational Players in Game Theory? finds GPT-4 moving from 10% to 100% in the Ring-Network game when forced to state its belief about the opponent before moving.
- SimpleToM's reminder of the model's own earlier mental-state answer restores Claude-3.5-Sonnet to 97.1%.
- Doing What They Say, Not What They Reason finds in poker that agents do execute their stated conclusions. The failure is upstream, in getting from reasoning to the conclusion, and a "apply the rule mechanically" instruction roughly halves rule misapplications.
Together these warn that a modular architecture presented as a fix may do no better than a one-line "state your belief first" prompt. We therefore treat modularity mainly as a measurement instrument. Any claim that it improves performance must beat that prompt.
Three systems supply the engineering. Shachi separates the environment interface from an agent made of an engine, configuration, memory and tools, and validates output format with a second parsing pass. It never checks whether actions follow the agent's beliefs. MultiAgentBench shows that good coordination talk can coexist with near-zero task scores (Llama-3.1-70B in Minecraft), which tells us to separate belief errors from execution errors. AgentSquare shows that module interfaces can be standardized and recombined, but its module library contains no social-belief modules.
Research Program
The architecture. We build the agent on a Shachi-style environment interface with three modules.
- A belief module reads the observation history and writes a typed belief record. Depending on the task, the record holds the partner's predicted next action, the partner's belief about a hidden fact, the partner's goal, or poker equity and pot odds as in Wang.
- An action module maps the record, plus a controlled amount of raw context, to a decision in a checkable format.
- A thin executor turns the decision into environment calls.
All modules use the same backbone model, so differences come from information flow, not capacity. The raw-context setting is a single dial with three positions: none, partner-visible observations only, or full history including privileged facts.
The belief-editing test. For each decision point we construct a counterfactual record $b'$ that differs from the original record $b$ in exactly one field. The edit is chosen so that the best response under $b'$, written $\pi^*(b')$, differs from the best response under $b$. In games $\pi^*$ is computed from the payoffs; in poker, from the equity rule. We define belief sensitivity as
$$S = \Pr\big[a(b') = \pi^*(b')\big],$$
where $a(b')$ is the action the action module emits after the edit. An agent with $S$ near 1 acts on its stated beliefs. An agent with $S$ near the rate of unchanged actions treats its stated beliefs as decoration.
We apply the same edit to the monolithic "state belief, then act" baseline by rewriting its stated-belief span in the prompt prefix and regenerating the action. This lets the test compare architectures on equal terms.
The error decomposition. Combining $S$ with belief accuracy (whether $b$ matches the ground truth the environment exposes) and execution checks, we split every wrong action into exactly one of three classes:
- Belief error: the record itself was wrong.
- Application error: the record was right, but the action did not follow it.
- Execution error: the decision was right, but the environment call failed.
This extends Wang's reasoning-to-conclusion-to-action split from closed-rule poker to multi-turn and communicative settings.
Mapping the tradeoff. We sweep the raw-context dial and also inject controlled noise into belief records, corrupting a known fraction $\epsilon$ of fields. For each setting we plot accuracy against $\epsilon$. Our hypothesis has two parts:
- With the belief record only, accuracy degrades roughly linearly in $\epsilon$, because every corrupted field passes through.
- With full context, accuracy stays flat in $\epsilon$ but sits lower when $\epsilon = 0$ on tasks with information asymmetry.
The two curves cross at a task-dependent point. That crossing point is the quantity we report.
Baselines
The naive reference. A single-call agent that sees the full context and answers directly, with no belief elicitation. It anchors every gap measurement. Reported numbers include 50% for GPT-4 on T4D-ToM zero-shot (FaR) and 10% for GPT-4 in the implicit Ring-Network setting (Fan et al.).
The strongest existing competitor: same-prompt belief elicitation. The model states its belief about the partner in a structured span and then acts, all in one call. Some versions also add SimpleToM's reminder of the model's earlier mental-state answer.
- It is the strongest competitor because it already reaches 100% on the Ring-Network (Fan et al.) and 97.1% for Claude-3.5-Sonnet on SimpleToM.
- It will run on the same backbones, the same episodes and seeds, the same token budget per decision, and an equal prompt-tuning budget on a shared development split.
- Beating it does not mean higher static accuracy, where it is already near ceiling. It means one of two things in the interactive tasks: a statistically significant improvement in regret or task success at matched compute, or higher belief sensitivity $S$ at equal accuracy.
Foresee-and-Reflect prompting. This is the published structured-prompt method. It reports 71.4% for GPT-4 on T4D-ToM, and 42% under noisy foresight. We run it with its published prompts under the same noise-injection protocol. That gives it a direct comparison on robustness to bad intermediate reasoning.
A frontier reasoning model. We include o1-preview or its successor, called monolithically. SimpleToM reports 84.1% behavior prediction and 59.5% judgment for o1-preview. This tests whether extra inference-time computation makes the modular split unnecessary.
The repeated-game reference. RMax, the tabular learner that Riemer et al. report outperforming LLMs. No reported regret number was found on Lacuna for RMax on these games, so we will measure it ourselves under the same partner policies and horizons.
The format-only control. A Shachi-style agent with two-stage parsing, to show what output-format enforcement alone buys.
The key ablations. These isolate what the modular split actually contributes:
- The same modules with the raw-context dial set to full, which removes the gating.
- The same pipeline with the belief record replaced by an empty field. If accuracy does not drop, the belief module is not doing the work.
Why not just…?
Why not just ask the model to state its belief before acting, in one prompt? This is the main threat, and on static tasks it probably wins. Fan et al. and SimpleToM both show that this one-line change closes most of the gap there. We do not claim otherwise. Our claim concerns multi-turn settings and measurement. In one prompt, the stated belief and the full privileged context sit side by side, so the context can override the belief. The prefix-editing variant of our test measures whether that happens. If the monolithic prompt matches the modular agent on both regret and $S$ across the repeated games and SOTOPIA, we will report that the architecture adds nothing beyond elicitation. The belief-editing test and the error decomposition would remain the contribution.
Why not just use a stronger reasoning model? In SimpleToM, o1-preview still scores 59.5% on judgment. In EnactToM, every evaluated frontier model scores 0.0% Pass^3 on hard tasks. Scale narrows the gap without closing it. We include such a model as a baseline, and the decomposition will show whether its remaining errors are belief errors or application errors.
Why not just have an LLM judge check that actions match reasoning? In a related faithfulness setting, an LLM judge caught 63.3% of drifted outputs against 89.6% for a formal check (The Faithfulness Gap). Pandey and Jajoo find that LLM reasons tend to echo surface features of the prompt. A judge reads correlation in text. Belief editing measures causal dependence, and we will report agreement between the judge and the editing test as a secondary result.
Why not just enforce a JSON schema, as Shachi does? Wang shows that the conclusion-to-action link already holds and that the failure is upstream. A schema checks form, not content. The format-only control quantifies this directly.
Proposed Experiments
Calibration on static tasks. We start with static tasks where ground truth is clean: SimpleToM and T4D-ToM, plus Faux-Pas, where FaR reports 76% for GPT-4 against 39% with standard chain of thought. These confirm that the elicitation baseline reaches its reported level in our setup.
The core experiments.
- The repeated games from Riemer et al. (Rock-Paper-Scissors, Iterated Battle of the Sexes, Iterated Prisoner's Dilemma) against fixed and tit-for-tat partners. Payoffs give exact best responses, so $S$ and regret per turn are exact.
- Fan et al.'s Ring-Network game, run over multiple rounds.
- Wang's poker harness, where equity defines the reference action.
Open-ended transfer. We then move to SOTOPIA. The belief record holds the partner's goal and whether a secret has been disclosed, and the edits target those fields. The outcome is Goal Completion; SOTOPIA-hard reports 4.85 for GPT-4 against 6.15 for humans. We also track the Secret dimension, since leaking private information is the curse-of-knowledge symptom we expect gating to reduce.
Decisive checks. Three outcomes would settle the questions:
- Whether the crossover in the tradeoff curve exists. If gating never helps at zero noise, the curse-of-knowledge account of interactive failure is wrong for these tasks.
- Whether $S$ is substantially below 1 for the monolithic elicitation baseline in multi-turn games, even when its static accuracy is near ceiling. This is the case where actions ignore stated beliefs that the model got right.
- Whether the share of belief errors grows with interaction length. That would locate the interactive gap upstream, as Wang found in poker, rather than in application.
Workshop authors to connect with
- Doing What They Say, Not What They Reason: Locating the Faithfulness Gap in LLM Agents: Yufeng Wang
- Reason-Mediated Behavioral Models for Auditing LLM Social Simulators: Atharva Pandey, Gautam Jajoo
References
- Reason-Mediated Behavioral Models for Auditing LLM Social Simulators
- SimpleToM: Exposing the Gap between Explicit ToM Inference and Implicit ToM Application in LLMs
- How FaR Are Large Language Models From Agents with Theory-of-Mind?
- Position: Theory of Mind Benchmarks are Broken for Large Language Models
- EnactToM: An Evolving Benchmark for Functional Theory of Mind in Embodied Agents
- Can Large Language Models Serve as Rational Players in Game Theory? A Systematic Analysis
- Doing What They Say, Not What They Reason: Locating the Faithfulness Gap in LLM Agents
- Reimagining Agent-based Modeling with Large Language Model Agents via Shachi
- MultiAgentBench : Evaluating the Collaboration and Competition of LLM agents
- AgentSquare: Automatic LLM Agent Search in Modular Design Space
- The Faithfulness Gap: Certifying Semantic Equivalence Between Natural-Language and Formal Mathematical Statements