Inferring Strategies from Observations in Long Iterated Prisoner's Dilemma Experiments

Eladio Montero-Porras, Jelena Grujic, Elias Fernández Domingos, Tom LenaertsarXiv 2022 · arXiv

Introduction

The Iterated Prisoner’s Dilemma (IPD) serves as a fundamental model for studying the evolution of cooperation. While theoretical strategies like "Tit-for-Tat" (TFT) or "Win-Stay-Lose-Shift" (WSLS) are well-understood, human behavior often deviates significantly from these mathematical ideals. Previous research has frequently been limited by short experimental durations—typically under 20 rounds—and the assumption that players choose from a fixed menu of strategies. This study addresses these gaps by employing a data-driven approach to infer strategies directly from human actions over 100 rounds, focusing on how the stability of partnerships influences strategic development.

Methodology

The researchers conducted 100-round IPD experiments with 188 participants across two primary treatments: Fixed Partners (FP), where pairs remained constant, and Shuffled Partners (SP), where opponents rotated. To decode the underlying strategies, a multi-stage analytical pipeline was used:

  1. Contextual Clustering: Players were grouped based on the frequency of game outcomes (CC, CD, DC, DD) they experienced.
  2. Behavioral Sub-clustering: Within context groups, players were further differentiated by their responses using K-means clustering and t-SNE visualization.
  3. HMM Inference: Hidden Markov Models (HMMs) were used to represent strategies as probabilistic logic, treating internal states as hidden and actions as emissions.
Figure 2: Comparison of context and cooperation probabilities between Fixed and Shuffled Partner treatments.

Key Insights: The Impact of Partnership Structure

The 25-Round Learning Curve

A critical finding was the existence of a "learning phase." Strategies in the first 25 rounds were found to be chaotic and inconsistent. This suggests that much of the existing literature, which often concludes before round 25, primarily captures transient learning behavior rather than stable strategic equilibria.

Behavioral Self-Organization vs. Strategic Entanglement

In the Fixed Partner treatment, the population exhibited "self-organization." Cooperators successfully identified one another and stabilized their behavior, while defectors did the same. This led to a polarized environment with distinct, well-separated strategic clusters.

Conversely, shuffling partners acted as a "noise injector." Without the ability to build a history with a specific individual, players remained "entangled" in a mixture of defensive and exploratory strategies. This prevented the emergence of a stable cooperative equilibrium, keeping the population in a state of strategic flux.

Results and Cluster Analysis

The analysis revealed that human strategies are often "messy" versions of theoretical models. For instance, human "WSLS-like" behavior frequently failed to include a signal to return to cooperation after mutual defection, making it less resilient to noise than its mathematical counterpart.

Figure 5: Contextual distributions for clusters A-F alongside a t-SNE visualization showing the separation of treatments and behavioral groups.

Quantitative results supported these observations:

  • Cooperation Rates: Cooperation was significantly higher in the FP treatment (77% after mutual cooperation) than in the SP treatment.
  • Cluster Quality: The FP treatment produced much more distinct behavioral clusters, with a silhouette score of approximately 0.68 for the cooperative cluster.
  • Strategy Complexity: Most human behavior could be modeled with simple 1-state or 2-state HMMs, indicating that while stochastic, the underlying logic is relatively reactive to the immediate past.

Discussion and Future Directions

While HMMs effectively capture reactive strategies, they currently rely on "memory-one" assumptions. Future research should explore longer-term memory and the impact of perception noise—where players might misinterpret an opponent's move. Additionally, extending this methodology to N-player dilemmas could provide deeper insights into group cooperation dynamics.

Summary updated

Abstract

While many theoretical studies have revealed the strategies that could lead to and maintain cooperation in the Iterated Prisoner's Dilemma, less is known about what human participants actually do in this game and how strategies change when being confronted with anonymous partners in each round. Previous attempts used short experiments, made different assumptions of possible strategies, and led to very different conclusions. We present here two long treatments that differ in the partner matching strategy used, i.e. fixed or shuffled partners. Here we use unsupervised methods to cluster the players based on their actions and then Hidden Markov Model to infer what are those strategies in each cluster. Analysis of the inferred strategies reveals that fixed partner interaction leads to a behavioral self-organization. Shuffled partners generate subgroups of strategies that remain entangled, apparently blocking the self-selection process that leads to fully cooperating participants in the fixed partner treatment. Analyzing the latter in more detail shows that AllC, AllD, TFT- and WSLS-like behavior can be observed. This study also reveals that long treatments are needed as experiments less than 25 rounds capture mostly the learning phase participants go through in these kinds of experiments.
BibTeX
@misc{monteroporras2022inferring,
  author = {Eladio Montero-Porras and Jelena Grujic and Elias Fernández Domingos and Tom Lenaerts},
  title = {{Inferring Strategies from Observations in Long Iterated Prisoner's Dilemma Experiments}},
  year = {2022},
  eprint = {2202.04171},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.04171},
}
Figure