Position: Theory of Mind Benchmarks are Broken for Large Language Models

Matthew Riemer, Zahra Ashktorab, Djallel Bouneffouf, Payel Das, Miao Liu, Justin D. Weisz, Murray Campbell

Abstract

Our paper argues that the majority of theory of mind benchmarks are broken because of their inability to directly test how large language models (LLMs) adapt to new partners. This problem stems from the fact that theory of mind benchmarks for LLMs are overwhelmingly inspired by the methods used to test theory of mind in humans and fall victim to a fallacy of attributing human-like qualities to AI agents. We expect that humans will engage in a consistent reasoning process across various questions about a situation, but this is known to not be the case for current LLMs. Most theory of mind benchmarks only measure what we call literal theory of mind: the ability to predict the behavior of others. However, this type of metric is only informative when agents exhibit self-consistent reasoning. Thus, we introduce the concept of functional theory of mind: the ability to adapt to agents in-context following a rational response to their behavior. We find that many open source LLMs are capable of displaying strong literal theory of mind capabilities, but seem to struggle with functional theory of mind -- even with exceedingly simple partner policies. Simply put, strong literal theory of mind performance does not necessarily imply strong functional theory of mind performance or vice versa. Achieving functional theory of mind, particularly over long interaction horizons with a partner, is a significant challenge deserving a prominent role in any meaningful LLM theory of mind evaluation.

The Human-Centric Fallacy in ToM Benchmarking

Theory of Mind (ToM)—the ability to attribute mental states to others—is a cornerstone of human social intelligence. As we deploy LLMs as personal assistants and agents, we need them to understand and adapt to human partners. However, current evaluation methods suffer from a "human-centric fallacy." In humans, we assume that if someone can predict a partner's move, they will use that information rationally. Current LLM benchmarks, such as the Sally-Anne false-belief test, are passive; they ask the model to explain a situation or predict an action. These benchmarks are fundamentally flawed because they ignore process consistency. An LLM might achieve a high score on a prediction task (Literal ToM) while remaining completely incapable of using that insight to guide its own behavior in a real interaction (Functional ToM).

Methodology: Interactive Game Theory

To address this, we move away from passive storytelling toward interactive game theory. Models are evaluated using repeated matrix games:

  • Rock, Paper, Scissors (RPS): A competitive game.
  • Iterated Battle of Sexes (IBS): A coordination game.
  • Iterated Prisoner’s Dilemma (IPD): A mixed-incentive game.

We formalize two primary metrics:

  1. Literal ToM Loss ($Δ_{ToM}/T$): How accurately the model predicts the partner's next move.
  2. Functional ToM Regret ($Δ_{Functional}/T$): The difference between the reward the model actually received and the reward it could have received if it had played optimally against that partner.

Key Results: The Prediction-Action Gap

Our evaluation reveals a massive gap where models can accurately guess what a partner will do but fail to adapt their own strategy to win. As shown in the comparison of various models, even high-performing models like Mixtral 8x7B exhibit significant functional regret despite near-perfect prediction accuracy.

Comparison of Literal and Functional ToM metrics across various LLMs.

A basic tabular reinforcement learning algorithm (RMax) consistently outperforms almost all LLMs in functional performance. For instance, while Mixtral achieved 96.7% prediction accuracy in RPS, its functional regret remained high (0.542) compared to the Tabular baseline (0.083). This highlights that strong literal ToM does not guarantee functional competence.

The Impact of Prompting Strategies

We tested several prompting strategies, including Chain of Thought (CoT), Reflexion, and Social Prompting. Surprisingly, social prompting—explicitly asking the model to predict the partner's move before choosing its own action—is not a silver bullet.

Performance of various prompting strategies across RPS, IBS, and IPD games.

As seen in the data across different games, models often correctly predict a partner's move but then fail to choose the rational counter-move. In RPS, for example, the Social LM prompt achieves 92.0% prediction accuracy, yet its functional regret (0.643) is nearly as high as standard prompting. This suggests a deep-seated reasoning deficit in translating social insights into strategic actions.

Insights and Future Directions

  • The Reasoning Paradox: Reasoning-heavy models like DeepSeek-R1-Distill-Qwen-32B show stronger functional performance even when their literal prediction accuracy is lower than simpler models. They seem to "know" how to win without necessarily articulating a perfect prediction.
  • Instruction Tuning Trade-offs: "Chat" versions of models generally performed better at functional tasks than base counterparts, likely because instruction tuning prioritizes interactive decision-making.
  • Oracle Failure: Even when explicitly told exactly what the partner would do (Oracle setting), many LLMs still failed to choose the rational counter-move.

Future research must move the industry toward Functional ToM evaluations that require models to actually play the game rather than just describe it from the sidelines. This includes testing adaptation to complex personas and maintaining consistent strategies over long-interaction horizons.

Summary updated

Abstract

Our paper argues that the majority of theory of mind benchmarks are broken because of their inability to directly test how large language models (LLMs) adapt to new partners. This problem stems from the fact that theory of mind benchmarks for LLMs are overwhelmingly inspired by the methods used to test theory of mind in humans and fall victim to a fallacy of attributing human-like qualities to AI agents. We expect that humans will engage in a consistent reasoning process across various questions about a situation, but this is known to not be the case for current LLMs. Most theory of mind benchmarks only measure what we call literal theory of mind: the ability to predict the behavior of others. However, this type of metric is only informative when agents exhibit self-consistent reasoning. Thus, we introduce the concept of functional theory of mind: the ability to adapt to agents in-context following a rational response to their behavior. We find that many open source LLMs are capable of displaying strong literal theory of mind capabilities, but seem to struggle with functional theory of mind -- even with exceedingly simple partner policies. Simply put, strong literal theory of mind performance does not necessarily imply strong functional theory of mind performance or vice versa. Achieving functional theory of mind, particularly over long interaction horizons with a partner, is a significant challenge deserving a prominent role in any meaningful LLM theory of mind evaluation.
BibTeX
@inproceedings{riemer2025position,
  author = {Matthew Riemer and Zahra Ashktorab and Djallel Bouneffouf and Payel Das and Miao Liu and Justin D. Weisz and Murray Campbell},
  title = {{Position: Theory of Mind Benchmarks are Broken for Large Language Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=BCP8UU2BcU},
}
Figure