Using Graph-Aware Reinforcement Learning to Identify Winning Strategies in Diplomacy Games (Student Abstract)

Hansin Ahuja, Lynnette Hui Xian Ng, Kokil JaidkaAAAI 2022 · DOI

Abstract

This research presents a two-tier framework that integrates sociolinguistic analysis with graph-aware Inverse Reinforcement Learning (IRL) to predict winners in the complex strategy game Diplomacy. By modeling both the persuasive language of player discourse and their structural influence within the game's social network, the proposed model achieves a 79% accuracy in identifying victors. Notably, the model demonstrates high predictive power even during the earliest stages of dialogue, suggesting that winning social dynamics are established early in the match.

Introduction

While artificial intelligence has achieved mastery in games with perfect information (e.g., Chess) or hidden information (e.g., Poker), the game of Diplomacy remains a unique challenge. Success in Diplomacy is predicated on negotiation, persuasion, and social coordination rather than just tactical board movements. This study addresses the gap in understanding how players influence others in high-stakes, multi-party environments by quantifying the advantage gained through linguistic choices and network positioning.

Methodology

The authors employ a two-tier methodology to capture the nuances of competitive discourse:

1. Sociolinguistic Strategy Labeling

The researchers identified four primary rhetorical strategies: Friendship (F), Reasoning (R), Game Move (GM), and Share Information (SI). To handle the large volume of unlabeled data in the CL-Aff Diplomacy dataset, a weakly supervised approach was used. Binary classifiers were trained on a labeled subset to categorize the remaining 15,000+ messages.

Performance of various classifiers across the four rhetorical strategy categories.

As shown in the classifier performance table, the C-SVC (Support Vector Classifier) proved most effective for identifying "Friendship" strategies (Macro-F1: 0.673), while Gaussian Naive Bayes was best suited for "Reasoning" (Macro-F1: 0.533).

2. Graph-Aware Score-Based Inverse Reinforcement Learning (SBIRL)

Inverse Reinforcement Learning (IRL) was used to infer the underlying reward function explaining player behavior. The core innovation is the inclusion of graph centrality features (e.g., authority and eigenvector centrality). This "graph-aware" approach accounts for a player's simultaneous engagement in multiple conversations and their overall influence within the communication network, rather than treating interactions in isolation.

Key Results and Discussion

The integration of network features significantly enhanced the model's predictive capabilities. The Graph-aware SBIRL model achieved an accuracy of 0.79, a substantial improvement over the context-agnostic SBIRL model's score of 0.71.

Accuracy of SBIRL vs. Graph-aware SBIRL as a function of the number of utterances, highlighting early-game kneepoints.

Analysis of the model's performance over time reveals a striking "early detection" capability. As illustrated in the accuracy curve, the model's performance stabilizes quickly. Using only the first half-dozen chat messages, the graph-aware model can predict the ultimate winner with high accuracy. This "kneepoint" analysis suggests that the rhetorical foundations for victory are often laid long before the final game state is reached.

Furthermore, the researchers identified a "Dichotomy of Persuasion": a strong anti-correlation exists between "Friendship" strategies and "Reasoning" or "Game Move" strategies. Players typically choose to either build rapport or utilize logical arguments, but rarely combine these approaches in a single utterance.

Conclusion and Future Work

This study demonstrates that a player's success in Diplomacy is a function of both what they say and who they are talking to. While the model shows high accuracy, the research is currently limited by a lack of counterfactual data—the inability to see how different rhetorical choices would have altered the game's outcome. Future work will focus on developing non-linear reward function estimators and generating synthetic counterfactual data to facilitate the creation of robust autonomous agents capable of complex social negotiation.

Summary updated

BibTeX
@inproceedings{ahuja2022using,
  author = {Hansin Ahuja and Lynnette Hui Xian Ng and Kokil Jaidka},
  title = {{Using Graph-Aware Reinforcement Learning to Identify Winning Strategies in Diplomacy Games (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21586},
}
Figure