Abstract
Cooperation plays a key role in the evolution of complex systems. However, the level of cooperation extensively varies with the topology of agent networks in the widely used models of repeated games. Here we show that cooperation remains rather stable by applying the reinforcement learning strategy adoption rule, Q-learning on a variety of random, regular, small-word, scale-free and modular network models in repeated, multi-agent Prisoner's Dilemma and Hawk-Dove games. Furthermore, we found that using the above model systems other long-term learning strategy adoption rules also promote cooperation, while introducing a low level of noise (as a model of innovation) to the strategy adoption rules makes the level of cooperation less dependent on the actual network topology. Our results demonstrate that long-term learning and random elements in the strategy adoption rules, when acting together, extend the range of network topologies enabling the development of cooperation at a wider range of costs and temptations. These results suggest that a balanced duo of learning and innovation may help to preserve cooperation during the re-organization of real-world networks, and may play a prominent role in the evolution of self-organizing, complex systems.
Introduction
Cooperation is the bedrock of complex biological and social systems, yet it remains evolutionarily puzzling: why help the group when defecting serves immediate self-interest? Previous research has demonstrated that the "topology" of a network—the specific map of interactions—heavily dictates whether cooperation survives. In "myopic" models, where agents simply imitate their most successful neighbor, cooperation often collapses if the network is slightly re-arranged. This sensitivity poses a problem for understanding the resilience of real-world networks that are constantly shifting. This study investigates strategy adoption rules that might render cooperation "topology-independent."
Methods
The researchers simulated multi-agent games, primarily the Prisoner’s Dilemma and Hawk-Dove games, across various network models including regular lattices, random networks, Watts-Strogatz "small-world" networks, Barabási-Albert "scale-free" networks, and modular networks. They compared standard "short-term" rules against "long-term" learning rules:
- Standard Rules (Short-term): "Best-takes-over" (imitate the neighbor with the highest payoff in the last round).
- Learning Rules (Long-term): Implementation of Q-learning, a reinforcement learning algorithm where agents maximize total discounted future rewards based on their own history.
- Innovation (Noise): A low level of randomness ($P_{innovation}$) was introduced, where an agent occasionally chooses the opposite of the "best" strategy to simulate errors or exploration.
The Fragility of Imitation
Under standard "imitate the winner" rules, cooperation is remarkably sensitive to structural changes. In one simulation, changing just 37 links out of 900 completely flipped the network from cooperative to defecting. This fragility highlights the limitations of simple imitation in maintaining social cohesion across diverse environments.
Learning and Innovation as Stabilizers
Long-term learning (Q-learning) allows agents to endure temporary losses for long-term gains. This "shadow of the past" helps cooperation survive temptation levels that would normally destroy it. However, pure learning does not fully solve topology sensitivity. The study identifies randomness (innovation) as a critical stabilizer. It prevents the network from "herding" into a sub-optimal state of total defection and allows cooperation to be re-introduced even if it was previously eliminated.
Key Results
- Stability: While "best-takes-over" rules saw cooperation levels swing wildly (from 3.5% to 18%) with minor network tweaks, Q-learning kept cooperation steady at approximately 39% regardless of rewiring.
- Temptation Resistance: In scale-free networks, Q-learning maintained stable cooperation even when the "temptation to defect" ($T$) was extremely high ($T > 4.5$), whereas standard rules saw cooperation drop to near zero once $T$ passed 3.7.
- Convergence: When innovation was added to long-term rules, the cooperation levels of vastly different networks converged, indicating that the strategy rule had become more influential than the network shape.
Conclusion
The combination of long-term memory and a low level of noise creates a "cooperation-compatible window" that fits almost any network structure. While these models are mathematically robust, future work is needed to determine if these "balanced duo" rules (learning + innovation) drive cooperation in specific biological or social systems, such as protein interaction networks or human organizations. Additionally, the role of dynamic topologies—where agents can move or choose partners—remains an open area for investigation.
Summary updated