Symmetric Machine Theory of Mind

Melanie Sclar, Graham Neubig, Yonatan BiskICML 2022 · OpenReview

Theory of Mind (ToM)—the ability to attribute mental states like knowledge, beliefs, and intentions to others—is a cornerstone of human cooperation. In the artificial intelligence community, most ToM research has historically been "asymmetric." In these setups, an agent either acts as a passive observer watching a video of someone else, or it is assigned a fixed, predefined role, such as a "listener" following a "speaker."

Real life, however, is rarely so rigid. We typically operate in symmetric environments where everyone can speak, listen, and move simultaneously. Existing AI models are rarely tested against these fluid dynamics. To bridge this gap, researchers have introduced SymmToM, a new multi-agent benchmark that moves ToM research away from passive observation and toward scenarios where every agent is an active, equal participant.

An example of symmetric Theory of Mind reasoning in the SymmToM environment.

The Challenge of Symmetric Environments

In symmetric roles, agents must coordinate while inferring. For example, if an agent sees two other agents meet in a distant corner, it must guess what information they likely exchanged so it doesn't waste time sharing redundant information with them later. The authors address the challenge of how agents can efficiently cooperate when they must constantly infer the changing knowledge states of their peers to achieve collective goals.

The SymmToM Environment

To test these dynamics, the paper introduces SymmToM, a procedurally generated grid-world environment evaluated across various configurations (e.g., $6\times6$ and $12\times12$ grids, with 3 or 4 agents).

  • The Goal: Agents must collect "information pieces" (represented as diamonds).
  • Global Vision, Local Hearing: Agents can see where everyone is on the map, but they can only hear messages from immediate neighbors.
  • Recharge Bases: To prevent the task from being solved too quickly, agents must periodically visit a base to "reset" and earn rewards. This creates a cycle of perpetual information seeking and introduces "information staleness." By wiping an agent's "second-hand" knowledge, the environment forces agents to constantly re-evaluate what others know now, rather than relying on outdated assumptions.
Illustration of the local communication range in the grid-world environment.

Explicitly Modeling Knowledge

The authors built their approach on top of MADDPG (Multi-Agent Deep Deterministic Policy Gradient), a standard framework for multi-agent reinforcement learning (RL). Their primary innovation is the explicit modeling of a knowledge matrix ($K$). Instead of hoping a neural network implicitly learns to track others, they designed agents that maintain a deliberate mental map of what every other agent likely knows.

They tested three specific inference strategies:

  1. Conservative (MADDPG-CE): Agents only update their model of others based on what they directly witness.
  2. Greedy (MADDPG-GE): Agents assume others are playing optimally and sharing the most valuable information possible when they meet.
  3. Estimated (MADDPG-EE): A differentiable approach using a neural network to estimate the probability of information exchange between agents outside of their own hearing range.

Key Findings and Results

  • ToM Models Win: Agents that explicitly modeled others' knowledge (the tailored $K$-matrix models) performed 42% better than vanilla recurrent models (RMADDPG).
  • The Importance of Memory: Standard RL agents suffer from "amnesia." Adding recurrence (memory) improved performance by roughly 60%, as it allowed agents to track past interactions.
  • The Oracle Gap: Despite improvements, there is still a massive performance gap (~127%) between standard agents and an "Oracle" agent that has perfect, god-like access to everyone's internal knowledge.
  • Heuristics vs. RL: Interestingly, a simple "round-robin" heuristic (move to the center, share everything, go to base) often outperformed the RL agents in complex settings. This suggests that current RL struggles with the high-dimensional spatial and social reasoning required for optimal play.
  • The "Sally-Anne" for Machines: The authors successfully adapted the classic child psychology "False Belief" test (Sally-Anne) into four ad-hoc tests to measure 0th, 1st, and 2nd-order Theory of Mind in their agents.

Limitations and Future Work

Modeling symmetric Theory of Mind remains an open challenge. The training time for these environments is exceptionally high; experiments with just 3-4 agents took thousands of hours of compute, making scalability a significant hurdle. Furthermore, while the models handled 1st-order ToM ("what you know") well, they struggled with 2nd-order ToM ("what you think they know").

Future work could introduce human baselines to provide a better "gold standard" than simple heuristics, or integrate natural language to see if the nuances of human speech make ToM easier or harder for machines to navigate.

Summary updated

Abstract

Theory of mind, the ability to model others’ thoughts and desires, is a cornerstone of human social intelligence. This makes it an important challenge for the machine learning community, but previous works mainly attempt to design agents that model the "mental state" of others as passive observers or in specific predefined roles, such as in speaker-listener scenarios. In contrast, we propose to model machine theory of mind in a more general symmetric scenario. We introduce a multi-agent environment SymmToM where, like in real life, all agents can speak, listen, see other agents, and move freely through the world. Effective strategies to maximize an agent’s reward require it to develop a theory of mind. We show that reinforcement learning agents that model the mental states of others achieve significant performance improvements over agents with no such theory of mind model. Importantly, our best agents still fail to achieve performance comparable to agents with access to the gold-standard mental state of other agents, demonstrating that the modeling of theory of mind in multi-agent scenarios is very much an open challenge.
BibTeX
@inproceedings{sclar2022symmetric,
  author = {Melanie Sclar and Graham Neubig and Yonatan Bisk},
  title = {{Symmetric Machine Theory of Mind}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=p7Rd9A3lPk},
}
Figure