Silly Rules Improve the Capacity of Agents to Learn Stable Enforcement and Compliance Behaviors

Raphael Koster, Dylan Hadfield-Menell, Gillian K. Hadfield, Joel Z. Leibo

Abstract

This study explores the mechanisms through which multi-agent reinforcement learning (MARL) populations develop stable social norms. We demonstrate that the introduction of "silly rules"—arbitrary taboos without direct functional utility—facilitates the learning of critical enforcement and compliance behaviors. By increasing the frequency of normative events, these arbitrary rules help agents overcome long-term credit-assignment challenges, leading to more robust social coordination and improved collective outcomes in environments with delayed environmental feedback.

Introduction

Human "ultrasociality" is fundamentally supported by third-party punishment, which ensures adherence to social norms. While many norms serve clear functional purposes (e.g., prohibiting theft or the consumption of toxins), human cultures are also replete with "silly rules"—arbitrary taboos with no immediate nutritional or survival benefit.

In the context of reinforcement learning, agents often struggle with credit assignment when rewards or penalties are significantly delayed. For instance, learning to avoid a poisonous substance that causes health depletion 100 timesteps later is a notoriously difficult task for standard RL architectures. We hypothesize that social norms, enforced via immediate third-party punishment, can bridge this temporal gap. Furthermore, we investigate whether adding arbitrary norms makes the normative landscape more "legible," providing agents with more frequent opportunities to practice the complex mechanics of social enforcement.

Methods

We utilized a partially-observable Markov game set in a 2D grid-world environment where agents forage for various berry types.

The 2D grid-world foraging environment. Agents (red/pink) navigate to collect berries while avoiding poisonous types. The yellow 'H' structure and visual markings indicate normative status.

One berry type is designated as "poisonous," reducing an agent's nutrient absorption after a 100-timestep delay. Agents are equipped with a "punishing beam" that imposes a penalty on a target at a small cost to the punisher. The environment implements a "normative classification scheme" by visually marking agents who violate a taboo. Punishing a marked agent yields a large reward, incentivizing third-party enforcement. We compared three experimental conditions:

  1. Norm-free: No rules or markings; agents must rely solely on delayed environmental feedback.
  2. Important rule: Only the consumption of poisonous berries results in marking.
  3. Silly rule: Consumption of either poisonous berries or a designated harmless berry results in marking.

Agents were trained using an IMPALA-style architecture with a convolutional visual encoder, an LSTM for memory, and a contrastive predictive coding (CPC) auxiliary loss.

Results and Analysis

Our results indicate that silly rules significantly enhance the learning process by increasing system legibility. By doubling the number of taboos, agents encounter rule violations more frequently, providing more training data for the complex sequence of actions required to aim and fire the punishing beam.

Performance metrics across training steps. (A) Misdirected punishment, (B) Total times punished, (C) Mean time marked, (D) Time since poisoning, (E) Taboo berries eaten, and (F) Collective return. The 'silly rule' condition (green) shows improved performance in intermediate stages.

Key findings include:

  • Hierarchy of Social Learning: Agents must first master the mechanics of punishment before the population can achieve compliance. A credible threat of enforcement is a prerequisite for norm adherence.
  • Solving Credit Assignment: Social norms transform a delayed-reward problem into an immediate social-feedback problem. Getting "zapped" immediately after eating a poisonous berry provides a clearer signal than getting sick much later.
  • Improved Compliance and Stability: Agents in the "silly rule" condition consistently consumed fewer poisonous berries than those in the "important rule" condition. Moreover, these populations were less prone to "indiscriminate punishing" (zapping innocent agents), suggesting more stable enforcement policies.
  • Collective Return: While the "silly rule" condition incurs a "dead-weight" cost by punishing harmless behavior, it achieves significantly higher collective returns during intermediate learning stages compared to the functional-only condition.

Discussion

The primary limitation of this work is that the normative classification scheme is predefined by the environment rather than emerging organically from agent interactions. While agents learn to enforce and comply, they do not yet "invent" the norms. Future research will explore the emergence of these patterns, allowing agents to dynamically define taboos and adapt social norms to shifting environmental pressures. In conclusion, arbitrary social structures can serve as a "scaffolding" that enables the learning of complex, functional social behaviors in multi-agent systems.

Summary updated

Abstract

Howcan societies learn to enforce and comply with social norms? Many if not most human norms are functional. Rules that punish non-cooperative behavior, for example, support cooperation. An intriguing feature of human normativity is that many social norms concern behaviors that have no direct impact on material wellbeing. Examples include rules about what color clothing one wears to a funeral [7] or whether one uses one's left or right hand in particular tasks [2]. Such apparently pointless rules are ubiquitous, often acquiring great social meaning despite the absence of functionality. Hadfield-Menell et al. (2019) call these norms "silly rules" and distinguish them from "important rules," such as rules that govern resource sharing or prohibit harmful conduct, that directly impact welfare [3].
BibTeX
@inproceedings{koster2020silly,
  author = {Raphael Koster and Dylan Hadfield-Menell and Gillian K. Hadfield and Joel Z. Leibo},
  title = {{Silly Rules Improve the Capacity of Agents to Learn Stable Enforcement and Compliance Behaviors}},
  booktitle = {AAMAS},
  year = {2020},
  url = {https://openreview.net/forum?id=HPaarSWZuJl},
}
Figure