Abstract
As AI agents increasingly operate in complex multi-agent environments, understanding their decision-making processes becomes critical. This paper introduces a novel framework for Explainable AI (XAI) specifically designed for "mixed-motive" games—scenarios where agents must balance cooperation and competition. By proposing three new explanation methods—Strategy-Based Utility Explanations (SBUE), Shared Interests Correlation Analysis (SICA), and Probable Actions—the authors demonstrate how these tools improve human understanding of strategic nuances and mitigate biases in Large Language Model (LLM) agents.
The Challenge of Mixed-Motive Games
Most existing XAI research focuses on single-agent systems or purely cooperative multi-agent environments. However, many real-world interactions are mixed-motive, where interests are partially aligned and partially opposed. In games like Diplomacy or Risk, an agent's decision is not merely about maximizing a score; it involves signaling intent, building trust, or anticipating betrayal.
Standard XAI methods, such as feature attribution, often fail in these settings because they do not capture the "social" or strategic logic of the game. For instance, a move in Diplomacy might appear sub-optimal in the short term but serves as a vital diplomatic signal to a neighbor. The authors identify a significant gap in our ability to account for inter-agent competition, non-binding "cheap-talk," and implicit communication through actions.
Proposed Explanation Framework
The researchers propose a three-level conceptual framework for explanations: Strategic (utility effects), Situational (environmental motivations), and Diplomatic (influence on future outcomes). To operationalize this, they developed three specific methods:
- Strategy-Based Utility Explanations (SBUE): Instead of showing only the agent's expected gain, SBUE presents a vector of expected utilities for all agents. This reveals whether a move is "friendly" (benefiting others) or "hostile" (harming others).
- Shared Interests Correlation Analysis (SICA): This method calculates the Pearson correlation between the utilities of pairs of agents. A high positive correlation suggests the environment favors cooperation, while a negative correlation indicates a zero-sum conflict.
- Probable Actions-Based Explanations: This uses simulations to show the most likely responses from other agents, providing an example-based look at immediate strategic consequences.
Experimental Setup and Environments
The framework was evaluated across three distinct environments to ensure generality:
- No-press Diplomacy: A 7-player game with a massive action space ($10^{21}$ to $10^{64}$ joint actions per turn), utilizing neural network policies.
- Communicate Out of Prison (COP): A 3-player game inspired by the Prisoner's Dilemma, featuring natural language communication between LLM agents (GPT-4) with distinct personalities.
- Risk: A simplified 4-player version of the classic board game using heuristic-based policies.
Key Results and Insights
The study yielded several significant findings regarding the effectiveness of these explanation methods:
- Human Alignment: In Diplomacy, SICA's ranking of "friends" and "enemies" aligned closely with expert human intuition, achieving a MAP@K score of 0.74-0.81.
- Improved Understanding: In the COP game, explanations improved users' understanding of inter-agent relationships by 59% and their understanding of diplomatic influence by 30%.
- Mitigating LLM Bias: The researchers found that LLM agents are often biased by the order of presented options (recency bias). Providing structured explanations helped mitigate this bias, leading to more rational decision-making.
- Convergence and Efficiency: Despite the massive state space of games like Diplomacy, the authors demonstrated that SICA and SBUE estimates converge reliably with a relatively small number of simulations (~200 samples).
Conclusion and Future Work
This research demonstrates that "diplomacy" and strategic intent can be interpreted through structured utility analysis just as clearly as through words. While the methods are computationally intensive, they provide a robust foundation for understanding complex agent interactions. Future work will explore extending these methods to "full-press" Diplomacy (including natural language negotiation) and investigating more complex association functions for SICA beyond Pearson's correlation.
Summary updated