Learning Fair Cooperation in Mixed-Motive Games with Indirect Reciprocity

Martin Smit, Fernando P. SantosIJCAI 2024

Abstract

In multi-agent systems, fostering cooperation among self-interested agents is a fundamental challenge, exacerbated when populations are divided into majority and minority groups. This paper investigates how Indirect Reciprocity (IR)—helping others based on their reputation—can sustain fair cooperation in such heterogeneous environments. Using Evolutionary Game Theory (EGT) and Reinforcement Learning (RL), the authors identify social norms and strategies that promote equitable outcomes, finding that while a defecting majority can suppress minority cooperation, robust norms like SternJudging can steer independent agents toward high-level, fair cooperation.

Introduction

In Multi-Agent Systems (MAS), achieving cooperation is a classic mixed-motive struggle. While collective cooperation yields the highest total benefit, individual agents are often incentivized to "free-ride" on the efforts of others. Indirect Reciprocity (IR) offers a powerful solution by allowing agents to build reputations: "Good" agents receive help, while "Bad" ones are denied.

Most existing research on IR assumes a homogeneous population. However, real-world populations are often divided by group identities, such as departments in a corporation or demographics in a marketplace. This research addresses a critical gap: how do group identities and potential social biases affect the stability and fairness of cooperation? If a majority group discriminates against a minority, can cooperation survive?

The Indirect Reciprocity Framework

The core of the study is the IR mechanism, where social norms dictate how reputations evolve based on actions.

The mechanism of Indirect Reciprocity (IR). Donors choose actions based on strategies and recipient reputations, which are updated according to social norms.

In this model, a donor decides whether to pay a cost $c$ to provide a benefit $b$ to a recipient. This decision is informed by the recipient's reputation and their group membership. A social norm then updates the donor's reputation based on their action and the recipient's prior status. For example, under the SternJudging norm, helping a "Good" person is viewed as "Good," while helping a "Bad" person is viewed as "Bad."

Methodology

The authors employed a dual modeling approach to capture both theoretical stability and practical learnability:

  1. Evolutionary Game Theory (EGT): The researchers performed an analytical stability analysis across $2^{16}$ combinations of norms and strategies. They sought to identify Evolutionarily Stable States (ESS)—configurations that can resist "invasion" by selfish mutant strategies.
  2. Reinforcement Learning (RL): To test if these stable states are reachable by independent agents, they modeled agents as tabular Q-learners. This allowed the study to account for stochastic effects and the learning process in a dynamic environment.

The population was split into a Majority (90%) and a Minority (10%). The study tested various social norms, including the "Leading Eight" norms from IR literature, such as SternJudging (SJ) and SimpleStanding (SS).

Key Insights: The Influence of Group Dynamics

A primary finding is the "dictatorship of the majority." The analysis revealed that if the majority group adopts an "Always Defect" strategy, there are zero stable states where the minority group continues to cooperate. The minority is essentially forced into selfishness by the majority's behavior. Conversely, a cooperative majority can often maintain its state even if the minority is selfish, though this results in a significant payoff disparity.

The research also highlights a counterintuitive relationship between norms and outcomes: "fair" norms (those that treat in-groups and out-groups identically) do not automatically guarantee fair outcomes. In some cases, "unfair" or biased norms can lead to high fairness if the agents' learned strategies provide a functional "counterweight" to the norm's bias.

Results: Fairness and Cooperativeness

The study identifies a small subset of norms and strategies that lead to both high cooperation and high fairness across both groups.

Fairness vs. cooperativeness across different social norms and strategies. Group-agnostic strategies often achieve high fairness and cooperation, whereas discriminatory strategies vary significantly.

SternJudging (SJ) emerged as the most robust norm for sustaining cooperation in heterogeneous groups. Its strictness in punishing defectors and rewarding those who punish "Bad" actors makes it highly resistant to exploitation.

In RL settings, the transition to these fair states was found to be highly sensitive to the agents' initial "optimism" (initial Q-values). If agents start with pessimistic expectations, the system almost inevitably collapses into universal defection. Furthermore, RL agents often required a higher benefit-to-cost ratio ($b/c$) to reach fair cooperation compared to what EGT analytical models suggested.

Conclusion and Future Work

This work demonstrates that while group divisions introduce significant challenges to fairness, carefully designed social norms can foster equitable cooperation among self-interested agents.

Limitations and Future Directions: * Dynamic Groups: Future models should consider agents who can change group membership or groups that emerge dynamically. * Private Reputations: Moving beyond "public" reputations to scenarios where agents have imperfect or conflicting information about others' status. * Norm Emergence: Investigating "bottom-up" norm emergence, where agents learn or evolve the social norms themselves, rather than having them imposed by a system designer.

Summary updated

BibTeX
@inproceedings{smit2024learning,
  author = {Martin Smit and Fernando P. Santos},
  title = {{Learning Fair Cooperation in Mixed-Motive Games with Indirect Reciprocity}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure