Abstract
This study explores the mechanisms through which multi-agent reinforcement learning (MARL) populations develop stable social norms. We demonstrate that the introduction of "silly rules"—arbitrary taboos without direct functional utility—facilitates the learning of critical enforcement and compliance behaviors. By increasing the frequency of normative events, these arbitrary rules help agents overcome long-term credit-assignment challenges, leading to more robust social coordination and improved collective outcomes in environments with delayed environmental feedback.
Introduction
Human "ultrasociality" is fundamentally supported by third-party punishment, which ensures adherence to social norms. While many norms serve clear functional purposes (e.g., prohibiting theft or the consumption of toxins), human cultures are also replete with "silly rules"—arbitrary taboos with no immediate nutritional or survival benefit.
In the context of reinforcement learning, agents often struggle with credit assignment when rewards or penalties are significantly delayed. For instance, learning to avoid a poisonous substance that causes health depletion 100 timesteps later is a notoriously difficult task for standard RL architectures. We hypothesize that social norms, enforced via immediate third-party punishment, can bridge this temporal gap. Furthermore, we investigate whether adding arbitrary norms makes the normative landscape more "legible," providing agents with more frequent opportunities to practice the complex mechanics of social enforcement.
Methods
We utilized a partially-observable Markov game set in a 2D grid-world environment where agents forage for various berry types.
One berry type is designated as "poisonous," reducing an agent's nutrient absorption after a 100-timestep delay. Agents are equipped with a "punishing beam" that imposes a penalty on a target at a small cost to the punisher. The environment implements a "normative classification scheme" by visually marking agents who violate a taboo. Punishing a marked agent yields a large reward, incentivizing third-party enforcement. We compared three experimental conditions:
- Norm-free: No rules or markings; agents must rely solely on delayed environmental feedback.
- Important rule: Only the consumption of poisonous berries results in marking.
- Silly rule: Consumption of either poisonous berries or a designated harmless berry results in marking.
Agents were trained using an IMPALA-style architecture with a convolutional visual encoder, an LSTM for memory, and a contrastive predictive coding (CPC) auxiliary loss.
Results and Analysis
Our results indicate that silly rules significantly enhance the learning process by increasing system legibility. By doubling the number of taboos, agents encounter rule violations more frequently, providing more training data for the complex sequence of actions required to aim and fire the punishing beam.
Key findings include:
- Hierarchy of Social Learning: Agents must first master the mechanics of punishment before the population can achieve compliance. A credible threat of enforcement is a prerequisite for norm adherence.
- Solving Credit Assignment: Social norms transform a delayed-reward problem into an immediate social-feedback problem. Getting "zapped" immediately after eating a poisonous berry provides a clearer signal than getting sick much later.
- Improved Compliance and Stability: Agents in the "silly rule" condition consistently consumed fewer poisonous berries than those in the "important rule" condition. Moreover, these populations were less prone to "indiscriminate punishing" (zapping innocent agents), suggesting more stable enforcement policies.
- Collective Return: While the "silly rule" condition incurs a "dead-weight" cost by punishing harmless behavior, it achieves significantly higher collective returns during intermediate learning stages compared to the functional-only condition.
Discussion
The primary limitation of this work is that the normative classification scheme is predefined by the environment rather than emerging organically from agent interactions. While agents learn to enforce and comply, they do not yet "invent" the norms. Future research will explore the emergence of these patterns, allowing agents to dynamically define taboos and adapt social norms to shifting environmental pressures. In conclusion, arbitrary social structures can serve as a "scaffolding" that enables the learning of complex, functional social behaviors in multi-agent systems.
Summary updated