Introduction
In social dilemmas such as the Prisoner’s Dilemma, a fundamental tension exists between individual rationality and collective welfare. While mutual cooperation yields the best group outcome, individuals are often incentivized to "defect" or free-ride on others' efforts. Traditional models, like those in Evolutionary Game Theory (EGT), suggest that spatial structures can foster cooperation through "spatial reciprocity"—where agents only interact with immediate neighbors. However, these models typically rely on simple imitation strategies.
This paper introduces a Multi-Agent Reinforcement Learning (MARL) framework where agents learn complex neighbor-selection behaviors through trial-and-error. By incorporating long-term memory and a dual-network architecture, agents learn to "ghost" defectors and cluster with cooperators, sustaining high cooperation levels even under high temptation to defect.
Methodology: Dual-Network Architecture
The proposed framework utilizes Deep Q-Learning (DQN) applied to a spatial Prisoner’s Dilemma on an $L \times L$ lattice. Each agent is controlled by two distinct neural networks that manage different aspects of the social interaction.
- Dilemma Policy Network ($\pi_{d_i}$): This network determines the agent's primary action: to Cooperate (C) or Defect (D).
- Interaction Selection Network ($\pi_{s_i}$): This network decides which neighbors in the von Neumann neighborhood to engage with. Crucially, an interaction only occurs if both agents mutually select each other, enabling a "selective interaction" mechanism.
The reward signal is a hybrid utility function combining standard personal payoff maximization with a social learning component based on a modified Fermi rule. Furthermore, agents utilize a weighted moving average of past payoffs, allowing them to build a "reputation" for their neighbors and make decisions based on long-term experience rather than immediate, noisy signals.
Key Insights: Ghosting and Self-Organization
The core mechanism driving cooperation in this model is the "Ghosting" effect. Selective interaction acts as a filter; agents learn to identify defectors and cease interactions with them. This deprives defectors of the exploitation payoffs they rely on, eventually incentivizing them to switch to cooperation to receive any reward.
The system undergoes an emergent self-organization process. Initially, defectors may invade (the END period), but as cooperators learn to cluster and exclude defectors (the EXP period), they form resilient, exclusive networks. Long-term memory ($\alpha=0.6$) is essential here, as it allows agents to distinguish between a reliable cooperator who had a single bad round and a persistent defector.
Results and Performance
The RL-based approach significantly outperforms traditional EGT baselines. While EGT models see a sharp decline in cooperation as the "temptation to defect" ($b$) increases, the MARL agents maintain near-total cooperation for much higher dilemma strengths.
Key findings include:
- Superior Resilience: The RL model maintains high cooperation up to $b=1.2$, whereas the EGT baseline drops to ~54% at $b=1.1$.
- Active Assortment: Agents actively avoid uncooperative pairings. "Cooperator-Cooperator" (CC) links formed at a rate of 77.29%, compared to only 45.48% for "Defector-Defector" (DD) links.
- Payoff Equality: Although the Gini coefficient initially spikes during the invasion of defectors, it drops significantly as selective interaction stabilizes the network, leading to more equitable reward distributions.
Conclusion and Future Work
This research demonstrates that autonomous agents can learn to sustain cooperation through selective interaction and long-term memory. However, limitations remain. The current model uses a fixed lattice topology, preventing agents from seeking new partners outside their immediate neighborhood. Additionally, the utility function relies on global information (average population payoff), which may not be available in fully decentralized settings. Future work could explore more complex social norms, varying levels of trust, and more flexible network topologies to further bridge the gap between MARL and real-world social dynamics.
Summary updated