Large Language Models (LLMs) frequently demonstrate a striking paradox: they can flawlessly deduce what a human is thinking, accurately predict an opponent's next move, or outline a perfect business strategy, yet completely fail to act on that knowledge when making a decision. This disconnect between internal reasoning and external behavior is known as the inference-action gap.
To understand why this gap exists, we must look at how language models process social and strategic information, and why descriptive fluency does not automatically translate into functional competence.
Literal vs. Functional Theory of Mind
The inference-action gap is most pronounced in social settings that require a "Theory of Mind" (ToM)—the ability to attribute mental states to others. Traditional benchmarks often evaluate LLMs by asking them to describe a character's beliefs. However, as Matthew Riemer et al. argue in Position: Theory of Mind Benchmarks are Broken for Large Language Models, these evaluations only measure literal Theory of Mind (predictive accuracy). They fail to assess functional Theory of Mind, which is the ability to adapt one's own behavior based on those predictions.
Consider a classic false-belief scenario: Alice thinks her keys are in the basket, but Bob has secretly moved them to the box. When asked "Where does Alice think the keys are?", an LLM will correctly answer "the basket." Yet, if the LLM is tasked with helping Alice find her keys, it will often tell her to look in the box.
[Scenario: Alice holds a false belief]
│
▼
┌────────────────────────────────────────────────────────┐
│ LLM Internal Inference │
│ "Alice incorrectly believes the keys are in the basket"│
└───────────────────────┬────────────────────────────────┘
│
┌───────────────┴───────────────┐
▼ ▼
┌──────────────────────────────┐ ┌──────────────────────────────┐
│ Literal ToM Task (Pass) │ │ Functional ToM Task (Fail) │
│ Q: "Where will Alice look?" │ │ Action: Help Alice find them│
│ A: "The basket." │ │ Output: "Look in the box!" │
└──────────────────────────────┘ └──────────────────────────────┘
This decoupling of explicit inference from implicit application is a systemic failure mode. In SimpleToM: Exposing the Gap between Explicit ToM Inference and Implicit ToM Application in LLMs, Yuling Gu et al. demonstrated that even when a model explicitly identifies a character's false belief, its subsequent predictions of behavior often revert to assuming the character has perfect information about the world.
To quantify this discrepancy, Pei Zhou et al. introduced the Thinking for Doing (T4D) benchmark in How FaR Are Large Language Models From Agents with Theory-of-Mind?. They found a steep performance drop when models transitioned from identifying mental states to selecting helpful interventions. Similarly, when evaluated in collaborative 3D environments via the EnactToM benchmark developed by Gurusha Juneja et al. in EnactToM: An Evolving Benchmark for Functional Theory of Mind in Embodied Agents, frontier models capable of verbally reporting a partner's beliefs failed almost entirely when required to use that information to coordinate real-time actions.
Strategic Planning vs. Tactical Execution
This gap is not unique to social psychology; it also degrades performance in game theory, long-term planning, and tool use.
In Can Large Language Models Serve as Rational Players in Game Theory? A Systematic Analysis, Caoyun Fan et al. observed that LLMs acting as game-theoretic players frequently fail to choose optimal actions even after correctly predicting their opponent's behavior. The model's internal reasoning chain concludes one thing, but its final action selection mechanism "overlooks" its own logical deductions.
This pattern repeats across diverse operational domains:
- Long-Term Business Planning: In YC-Bench: Benchmarking AI Agents for Long-Term Planning and Consistent Execution, Muyu He et al. noted that simulated startup agents could explicitly identify a rule or strategic constraint within their internal reasoning chain, yet immediately violate that exact rule when emitting their final decision token.
- Scientific Automation: Evaluating multi-agent protein design, Anda-Raluca Epure et al. documented a reliability gap in Minibinder Lab: The Reliability Gap Of Agents For Designing High Quality Protein Binders. The agent would textually explain exactly which biochemical parameter needed adjustment, but then fail to update the corresponding tool parameters in its executable code block.
The Misalignment of Intent and Execution
When agents attempt to manifest their intentions in interactive environments, the gap reveals itself as a form of behavioral hypocrisy: the agent "talks the talk" but cannot "walk the walk."
For instance, in conversational games like Avalon, Ziyi Liu et al. showed in InterIntent: Investigating Social Intelligence of LLMs via Intention Understanding in an Interactive Game Context that models can successfully select a high-level strategic social intention (e.g., "convince the group I am innocent"). However, they struggle to generate dialogue that actually realizes that goal, often lapsing into vague or contradictory phrasing that undermines their strategy.
When moving from dialogue to physical or simulated sandboxes, the execution demands multiply. As Chenxu Wang 0001 et al. demonstrated in Towards Objectively Benchmarking Social Intelligence of Language Agents at the Action Level, agents frequently express a clear social commitment in text, but fail to execute the sequence of spatial and temporal actions required to fulfill it within a simulation.
This is reminiscent of foundational challenges in robotics identified by Chandrakant Bothe et al. in Towards Dialogue-based Navigation with Multivariate Adaptation driven by Intention and Politeness for Social Robots, where a robot might understand what a user wants linguistically, but fail to translate the sociolinguistic nuances of the request (like a user's impatience) into physical execution parameters like walking speed or vocal pitch.
Why the Gap Persists: An Intuitive View
To understand why this gap occurs, we must look at how autoregressive language models generate text.
When an LLM is prompted to reason, it generates tokens that navigate a semantic space optimized for description. Writing out a logical deduction or identifying a character's belief relies on statistical correlations found in explanatory text.
However, generating the subsequent action token requires navigating a different conditional probability distribution. The model does not possess an explicit cognitive architecture that forces its action-selection tokens to be causally bound by the logical constraints of its reasoning tokens. Instead, the model treats "reasoning" and "acting" as sequential text-generation tasks.
If the transition from reasoning text to action text is underrepresented in the training data, or if the context window becomes cluttered with distracting details, the statistical weight of the reasoning tokens weakens. The model effectively "forgets" how to apply its own deductions, leading to a breakdown where high-level cognitive understanding fails to guide behavioral execution.
Key Foundational Work
How FaR Are Large Language Models From Agents with Theory-of-Mind? (2023) — contributed 3 concept elements to this direction.
Towards Dialogue-Based Navigation with Multivariate Adaptation Driven by Intention and Politeness for Social Robots (2018) — contributed 1 concept elements to this direction.