To Promote Full Cooperation in Social Dilemmas, Agents Need to Unlearn Loyalty

Chin-wing Leung, Tom Lenaerts, Paolo Turrini

Abstract

This paper investigates how autonomous agents learn partner-selection rules in social dilemmas like the Prisoner’s Dilemma. Using multi-agent reinforcement learning, the authors demonstrate that while agents naturally learn to "ostracize" defectors, a lingering sense of "loyalty"—staying with a partner regardless of their behavior—acts as a primary bottleneck preventing full cooperation. The study reveals that for cooperation to flourish, agents must unlearn this loyalty, a process heavily dependent on the speed of network rewiring relative to strategy imitation.

Introduction

In evolutionary game theory, cooperation remains a puzzle. In well-mixed populations, "defectors" who exploit others typically dominate. Network reciprocity offers a solution: if cooperators can choose their partners, they can form protective clusters. While previous research often relied on hand-coded rewiring rules, this study asks whether agents can learn these rules autonomously through reinforcement learning and what specific behaviors emerge during this process.

Methodology

The researchers developed a co-evolutionary model where game strategies (Cooperate vs. Defect) and network structure (who is linked to whom) evolve simultaneously.

  • Game Dynamics: Agents interact in various social dilemmas, including the Prisoner’s Dilemma (PD), Stag Hunt (SH), and Snow-Drift (SD).
  • Link Learning: Agents use Multi-agent Q-learning to decide whether to "Stay" with or "Leave" a partner after observing their strategy.
  • Strategy Update: Agents update their game strategy by imitating more successful neighbors via a probabilistic Fermi function.
  • Rewiring Speed ($W$): A critical parameter representing the frequency of link updates relative to strategy updates.

The simulations involved a population of 1,000 agents initially arranged in a random $z$-regular graph, running for 1,000,000 iterations to ensure convergence.

Results and Discussion

The Emergence of "Out-for-Tat"

Agents naturally learn an "Out-for-Tat" (OFT) policy: maintaining links with cooperators and severing ties with defectors. This confirms that partner selection is a robust, learnable mechanism for sustaining cooperation without pre-programmed rules.

The Loyalty Trap and Rewiring Speed

A significant finding is the "loyalty" trap, where agents adopt a "Stay" strategy despite a partner's defection. This often occurs because the risk of rewiring to an unknown, potentially worse partner outweighs the known cost of the current defector. Overcoming this requires a high rewiring speed ($W$).

Cooperation levels across the T-S parameter space for varying rewiring speeds (W). Increasing W expands the blue regions of full cooperation.

As shown in Figure 4, as the rewiring speed $W$ increases, the population reaches higher levels of cooperation across a broader range of game parameters. In the Prisoner’s Dilemma and Stag Hunt, full cooperation is achievable if $W$ is sufficiently high. However, in the Snow-Drift game, cooperation levels often plateau around 80% because the incentive to leave a defector is lower.

Network Density and Policy Dominance

Network density ($z$) also plays a crucial role. Higher density makes cooperation harder to achieve because cooperators must prune more defectors from their social circles, requiring even higher rewiring speeds to succeed.

Impact of rewiring speed (W) and network density (z) on cooperation levels (top) and the distribution of learned policies (bottom). As W increases, the "Stay" policy (loyalty) is replaced by the dominant "OFT" policy.

Figure 6 highlights the transition from loyalty to effective partner selection. The bottom panel demonstrates that as $W$ increases, the percentage of the population adopting the "Stay" policy drops sharply, while the "OFT" policy becomes dominant, eventually exceeding 90% in successful populations.

Conclusion

This research demonstrates that while agents can learn to cooperate through partner selection, "loyalty" to defectors is a major psychological and algorithmic hurdle. Achieving full cooperation requires agents to "unlearn" this loyalty by prioritizing rapid network restructuring. Future work should explore more complex game strategies beyond simple imitation and investigate how these learned rules align with actual human behavior in social networks.

Summary updated

BibTeX
@inproceedings{leung2024promote,
  author = {Chin-wing Leung and Tom Lenaerts and Paolo Turrini},
  title = {{To Promote Full Cooperation in Social Dilemmas, Agents Need to Unlearn Loyalty}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure