Explaining Decisions of Agents in Mixed-Motive Games

Maayan Orner, Oleg Maksimov, Akiva Kleinerman, Charles Ortiz, Sarit Kraus

Abstract

As AI agents increasingly operate in complex multi-agent environments, understanding their decision-making processes becomes critical. This paper introduces a novel framework for Explainable AI (XAI) specifically designed for "mixed-motive" games—scenarios where agents must balance cooperation and competition. By proposing three new explanation methods—Strategy-Based Utility Explanations (SBUE), Shared Interests Correlation Analysis (SICA), and Probable Actions—the authors demonstrate how these tools improve human understanding of strategic nuances and mitigate biases in Large Language Model (LLM) agents.

The Challenge of Mixed-Motive Games

Most existing XAI research focuses on single-agent systems or purely cooperative multi-agent environments. However, many real-world interactions are mixed-motive, where interests are partially aligned and partially opposed. In games like Diplomacy or Risk, an agent's decision is not merely about maximizing a score; it involves signaling intent, building trust, or anticipating betrayal.

Standard XAI methods, such as feature attribution, often fail in these settings because they do not capture the "social" or strategic logic of the game. For instance, a move in Diplomacy might appear sub-optimal in the short term but serves as a vital diplomatic signal to a neighbor. The authors identify a significant gap in our ability to account for inter-agent competition, non-binding "cheap-talk," and implicit communication through actions.

Proposed Explanation Framework

The researchers propose a three-level conceptual framework for explanations: Strategic (utility effects), Situational (environmental motivations), and Diplomatic (influence on future outcomes). To operationalize this, they developed three specific methods:

  • Strategy-Based Utility Explanations (SBUE): Instead of showing only the agent's expected gain, SBUE presents a vector of expected utilities for all agents. This reveals whether a move is "friendly" (benefiting others) or "hostile" (harming others).
  • Shared Interests Correlation Analysis (SICA): This method calculates the Pearson correlation between the utilities of pairs of agents. A high positive correlation suggests the environment favors cooperation, while a negative correlation indicates a zero-sum conflict.
  • Probable Actions-Based Explanations: This uses simulations to show the most likely responses from other agents, providing an example-based look at immediate strategic consequences.
Figure 1: Visualization of SBUE and SICA metrics on a Diplomacy game board, showing how different countries' utilities are impacted by a specific move.

Experimental Setup and Environments

The framework was evaluated across three distinct environments to ensure generality:

  1. No-press Diplomacy: A 7-player game with a massive action space ($10^{21}$ to $10^{64}$ joint actions per turn), utilizing neural network policies.
  2. Communicate Out of Prison (COP): A 3-player game inspired by the Prisoner's Dilemma, featuring natural language communication between LLM agents (GPT-4) with distinct personalities.
  3. Risk: A simplified 4-player version of the classic board game using heuristic-based policies.

Key Results and Insights

The study yielded several significant findings regarding the effectiveness of these explanation methods:

  • Human Alignment: In Diplomacy, SICA's ranking of "friends" and "enemies" aligned closely with expert human intuition, achieving a MAP@K score of 0.74-0.81.
  • Improved Understanding: In the COP game, explanations improved users' understanding of inter-agent relationships by 59% and their understanding of diplomatic influence by 30%.
  • Mitigating LLM Bias: The researchers found that LLM agents are often biased by the order of presented options (recency bias). Providing structured explanations helped mitigate this bias, leading to more rational decision-making.
  • Convergence and Efficiency: Despite the massive state space of games like Diplomacy, the authors demonstrated that SICA and SBUE estimates converge reliably with a relatively small number of simulations (~200 samples).
Figure 4: Convergence analysis showing that SICA (Average Cosine Similarity) and SBUE (RMSE) stabilize as the sample size increases toward 250.

Conclusion and Future Work

This research demonstrates that "diplomacy" and strategic intent can be interpreted through structured utility analysis just as clearly as through words. While the methods are computationally intensive, they provide a robust foundation for understanding complex agent interactions. Future work will explore extending these methods to "full-press" Diplomacy (including natural language negotiation) and investigating more complex association functions for SICA beyond Pearson's correlation.

Summary updated

Abstract

In recent years, agents have become capable of communicating seamlessly via natural language and navigating in environments that involve cooperation and competition, a fact that can introduce social dilemmas. Due to the interleaving of cooperation and competition, understanding agents' decision-making in such environments is challenging, and humans can benefit from obtaining explanations. However, such environments and scenarios have rarely been explored in the context of explainable AI. While some explanation methods for cooperative environments can be applied in mixed-motive setups, they do not address inter-agent competition, cheap-talk, or implicit communication by actions. In this work, we design explanation methods to address these issues. Then, we proceed to establish generality and demonstrate the applicability of the methods to three games with vastly different properties. Lastly, we demonstrate the effectiveness and usefulness of the methods for humans in two mixed-motive games. The first is a challenging 7-player game called no-press Diplomacy. The second is a 3-player game inspired by the prisoner's dilemma, featuring communication in natural language.
BibTeX
@inproceedings{orner2025explaining,
  author = {Maayan Orner and Oleg Maksimov and Akiva Kleinerman and Charles Ortiz and Sarit Kraus},
  title = {{Explaining Decisions of Agents in Mixed-Motive Games}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i22.34493},
}
Figure