Can Large Language Models Adapt to Other Agents In-Context?

Matthew Riemer, Zahra Ashktorab, Djallel Bouneffouf, Payel Das, Miao Liu, Justin D. Weisz, Murray Campbell

Abstract

This position paper argues that the majority of theory of mind benchmarks are broken because of their inability to directly test how large language models (LLMs) adapt to new partners. This problem stems from the fact that theory of mind benchmarks for LLMs are overwhelmingly inspired by the methods used to test theory of mind in humans and fall victim to a fallacy of attributing human-like qualities to AI agents. We expect that humans will engage in a consistent reasoning process across various questions about a situation, but this is known to not be the case for current LLMs. Most theory of mind benchmarks only measure what we call literal theory of mind: the ability to predict the behavior of others. Measuring this kind of reasoning is very informative in testing the ability of agents with self-consistent reasoning. However, it is important to note the distinction between this and what we actually care about when this self-consistency cannot be taken for granted. We call this functional theory of mind: the ability to adapt to agents in-context following a rational response to predictions about their behavior. We find that top performing open source LLMs may display strong capabilities in literal theory of mind, depending on how they are prompted, but seem to struggle with functional theory of mind -- even when partner policies are exceedingly simple. Simply put, strong literal theory of mind performance does not necessarily imply strong functional theory of mind performance. Achieving functional theory of mind, particularly over long interaction horizons with a partner, is a significant challenge deserving a prominent role in any meaningful LLM theory of mind evaluation.

Introduction: The Literal vs. Functional Theory of Mind

Theory of Mind (ToM)—the ability to attribute mental states to others—is a cornerstone of human social intelligence. In AI research, ToM is typically evaluated using passive, question-answering tasks like the Sally-Anne test. However, these benchmarks often fall victim to the fallacy of attributing human-like qualities to AI. In humans, we assume that if someone can predict a partner's behavior, they will use that knowledge to inform their own actions.

This paper demonstrates that for LLMs, this "process consistency" does not exist. The authors distinguish between two types of reasoning:

  • Literal ToM: The ability to predict what another agent will do.
  • Functional ToM: The ability to adapt one's own behavior rationally based on those predictions.

Current LLMs can appear brilliant at Literal ToM while remaining functionally "blind" in interactive settings, making existing benchmarks misleading for evaluating real-world social or collaborative performance.

Methodology: Interactive Matrix Games

To move beyond passive Q&A, researchers used repeated matrix games as a controlled laboratory for interaction. These games included Rock, Paper, Scissors (RPS), Iterated Battle of Sexes (IBS), and Iterated Prisoner’s Dilemma (IPD). LLMs played 100-round matches against various partner policies, ranging from simple "Single Action" agents to more complex "Tit-for-Tat" strategies.

Performance was measured using functional regret ($\Delta_{Functional}/T$), which quantifies how much potential reward the model missed, and prediction error ($\Delta_{ToM}/T$). A wide range of models and prompting strategies—including Chain of Thought (CoT), Social Prompting, and Oracle Prompting—were evaluated against a Tabular RMax reinforcement learning baseline.

Results and Analysis

The Functional ToM Gap

The most striking finding is the massive gap between prediction and action. As shown in the model comparison for Rock, Paper, Scissors, models like Mixtral 8x7B can predict an opponent's move with over 96% accuracy but still fail to choose the winning counter-move, resulting in high functional regret compared to the Tabular baseline.

Comparison of LLM performance against a Tabular baseline on Rock, Paper, Scissors (RPS).

Impact of Prompting and Game Dynamics

Prompting strategies significantly influence performance, but rarely close the functional gap. While "Social Prompting" (forcing the model to predict before acting) can improve Literal ToM, models often ignore their own written predictions when selecting an action. Furthermore, even when provided with the ground-truth next move of the partner ("Oracle Prompting"), models struggle to calculate the optimal response over long horizons.

Performance breakdown across different games and prompting strategies, highlighting the persistent gap between prediction accuracy (ToM %) and functional regret.

Key insights from the results include:

  • Instruction Tuning Trade-offs: "Chat" versions of models often show worse Literal ToM than base models but better Functional ToM, suggesting tuning prioritizes interactive decision-making.
  • Inductive Bias: Canonical names like "Rock, Paper, Scissors" provide an initial boost due to pre-training but can hinder long-term adaptation compared to neutral labels.
  • Reasoning Models: Newer reasoning-focused models like DeepSeek-R1 show stronger Functional ToM, sometimes even bypassing explicit literal prediction to act rationally.

Conclusion

If LLMs fail to adapt in simple matrix games, they are likely even more limited in complex social nuances. This study highlights that strong literal theory of mind performance does not imply functional competence. Future ToM evaluations must shift toward interactive benchmarks that directly measure an agent's ability to act on its social understanding across diverse contexts.

Summary updated

Abstract

This position paper argues that the majority of theory of mind benchmarks are broken because of their inability to directly test how large language models (LLMs) adapt to new partners. This problem stems from the fact that theory of mind benchmarks for LLMs are overwhelmingly inspired by the methods used to test theory of mind in humans and fall victim to a fallacy of attributing human-like qualities to AI agents. We expect that humans will engage in a consistent reasoning process across various questions about a situation, but this is known to not be the case for current LLMs. Most theory of mind benchmarks only measure what we call literal theory of mind: the ability to predict the behavior of others. Measuring this kind of reasoning is very informative in testing the ability of agents with self-consistent reasoning. However, it is important to note the distinction between this and what we actually care about when this self-consistency cannot be taken for granted. We call this functional theory of mind: the ability to adapt to agents in-context following a rational response to predictions about their behavior. We find that top performing open source LLMs may display strong capabilities in literal theory of mind, depending on how they are prompted, but seem to struggle with functional theory of mind -- even when partner policies are exceedingly simple. Simply put, strong literal theory of mind performance does not necessarily imply strong functional theory of mind performance. Achieving functional theory of mind, particularly over long interaction horizons with a partner, is a significant challenge deserving a prominent role in any meaningful LLM theory of mind evaluation.
BibTeX
@inproceedings{riemer2025can,
  author = {Matthew Riemer and Zahra Ashktorab and Djallel Bouneffouf and Payel Das and Miao Liu and Justin D. Weisz and Murray Campbell},
  title = {{Can Large Language Models Adapt to Other Agents In-Context?}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=BCP8UU2BcU},
}
Figure