TL;DR
Current Theory of Mind (ToM) benchmarks for Large Language Models are fundamentally broken because they only evaluate a model's ability to predict others' behavior ("literal ToM"), while ignoring whether it can actually adapt its own actions accordingly ("functional ToM"). By testing LLMs in simple repeated games like Rock-Paper-Scissors, the authors reveal a massive "process consistency" gap: models can accurately predict an opponent's next move with near-perfect accuracy yet completely fail to select the rational counter-action.
What Problem Does This Paper Address?
Theory of Mind—the cognitive capacity to attribute mental states, beliefs, and desires to oneself and others—is a holy grail for creating socially intelligent AI. To date, many studies claim LLMs possess human-like ToM based on passive question-answering benchmarks (like the classic Sally-Anne false-belief test).
However, the authors point out a critical fallacy: when evaluating humans, we take for granted that the ability to predict someone else's behavior will be consistently applied to our own decision-making. LLMs, optimized purely for next-token prediction, lack this inherent "process consistency." They can generate a compelling explanation of an opponent's strategy but fail to act on it. This paper addresses this gap by formalizing and evaluating Functional Theory of Mind—the ability to adapt to a partner's policy in-context to maximize interactive success.
Methods
The authors disentangle ToM evaluation into two distinct mathematical formulations:
- Literal ToM Loss ($L_{\text{Literal}}$): Measures the distance between the model's prediction of an opponent's actions and the opponent's actual actions.
- Functional ToM Regret ($\Delta_{\text{Functional}}$): Measures the difference in expected reward between the model's chosen actions and the truly optimal response to the opponent's policy.
To evaluate these metrics, they place various open-source LLMs (including the LLaMA-2/3 families, Mixtral, and Mistral Large) into repeated game-theoretic environments over long interaction horizons (100 steps). They test multiple prompting strategies:
- Generic Strategies: Standard Language Modeling (LM) prompting and Question-Answering (QA) prompting.
- In-Context RL Strategies: Chain-of-Thought (CoT), Reflexion (maintaining a verbal memory of past mistakes), and Plans+Insights prompting.
- Explicit ToM Strategies: "Social Prompting," where the model is forced to explicitly predict the opponent's action before generating its own move.
- Oracle Prompting: A control setup where the opponent's exact next action is fed directly into the prompt as ground truth.
Datasets
Rather than standard text datasets, the authors evaluate models using three canonical game-theoretic matrix environments with controlled partner policies:
- Rock, Paper, Scissors (RPS): A purely competitive game.
- Iterated Battle of the Sexes (IBS): A coordination/cooperative game.
- Iterated Prisoner's Dilemma (IPD): A mixed-incentive game.
The models play against simple, out-of-distribution partner policies, such as a static single-action policy (e.g., an opponent that always plays "Rock") and dynamic "Tit-for-Tat" policies.
Key Insights and Ideas
- The Illusion of Competence: LLMs display remarkably high literal ToM but abysmal functional ToM. For instance, against an opponent playing only "Rock," an LLM quickly learns to predict "Rock" with near 100% accuracy, yet its own actions remain split evenly between Rock, Paper, and Scissors (the Nash equilibrium strategy, which optimizes for the worst-case opponent rather than exploiting the current one).
- Oracle Failure: Even when given "Oracle" inputs—where the prompt explicitly states exactly what move the opponent will make next—LLMs still fail to choose the optimal response over long context windows. This suggests the bottleneck isn't just a failure of prediction, but a deeper inability to reason over long interaction histories and payoff matrices.
- The Reasoning Model Twist: When testing newer, reinforcement-learning-trained reasoning models like DeepSeek-R1, the authors observed a bizarre inversion: the model achieved excellent functional ToM (low regret) but had worse literal ToM prediction accuracy. This implies that optimizing directly for reasoning chains can actually suppress explicit predictive accuracy while improving behavioral outcomes.
Key Results
Across the board, standard LLMs fail to match the performance of a simple, non-neural tabular baseline (the RMax reinforcement learning algorithm).
As shown in Table 1, models like LLaMA-2 70B achieve over 96% prediction accuracy (ToM %) but suffer a functional regret per step ($\Delta_{\text{Functional}}/T$) close to 1.0, meaning they perform no better than random guessing when it comes to maximizing reward.
Table 2 demonstrates that while advanced prompting techniques like Chain-of-Thought (CoT) and Social Prompting reduce functional regret compared to basic QA prompting, a significant performance gap remains when compared to the tabular baseline.
Table 3 highlights that even with Oracle inputs, LLaMA-3 70B Instruct consistently underperforms the tabular model, illustrating the severe difficulty LLMs face when reasoning over long contexts and complex payoff structures.
Limitations and Future Work
- Simplicity of Environments: The study relies on simple matrix games with static or basic dynamic partners. The authors note that future benchmarks must include more complex, interactive language games like Codenames, Hanabi, or Taboo.
- In-Context Learning Bottlenecks: Adapting to partners purely via in-context learning over long histories is computationally expensive and fragile. The authors suggest that parameter-efficient fine-tuning (PEFT) or model-merging techniques might be required for true, robust user adaptation in real-world deployments.
Summary updated