Mechanism Design Is Not Enough: Prosocial Agents for Cooperative AI

Xuanqiang Angelo Huang, Charlie Tharas, Samuele Marro, Van Q. Truong, Bernhard Schölkopf, Emanuele La Malfa, Zhijing Jin

TL;DR

This paper argues that designing perfect rules and incentives (mechanism design) is mathematically insufficient to ensure safe, cooperative behavior among AI agents. Because real-world environments are too complex for "complete" contracts, the authors prove that agents must be built with intrinsic prosociality—a direct concern for the welfare of others—to avoid inevitable social welfare loss.

What Problem Does This Paper Address?

As AI agents become more autonomous, they will increasingly interact in shared environments—negotiating service agreements, trading information, or managing resources. Traditionally, researchers have looked to mechanism design (a field of economics) to solve this. The idea is to create "rules of the game" (like auctions or contracts) so that even selfish agents produce a socially desirable outcome.

However, the authors point out a fundamental flaw: contractual incompleteness. In complex, open-ended worlds, it is impossible to write a rule for every possible future contingency. When agents encounter "incontractible" situations—where their actions can't be fully observed or enforced by the mechanism—self-interested agents will inevitably default to sub-optimal, non-cooperative behavior (like the Prisoner’s Dilemma). The paper asks: Can we bridge this "cooperation gap" by changing the agents themselves rather than just the rules?

Methods

The researchers combined formal game theory with large-scale LLM simulations:

  1. Theoretical Framework: They used incomplete contract theory to define "incontractible cells"—sets of future states that a mechanism cannot distinguish. They formally proved (Theorem 3.4) that in these cells, self-interested agents suffer a strictly positive welfare loss that no realistic mechanism can eliminate.
  2. $\lambda$-Cooperation Model: They modeled prosociality using a parameter $\lambda$, where an agent’s utility is its own payoff plus $\lambda$ times the total social welfare.
  3. LLM Simulations: They tested five frontier models (including the GPT-5.4 family, GPT-4o, and Grok-4.1) across three contracting regimes:
  4. No Contract: Simple "cheap talk" communication.
  5. Natural Language (NL) Contract: Agents vote on an informal, unenforced agreement.
  6. Code Contract ("Code Law"): Agents negotiate a contract, and a "coder agent" implements it as Python code that is strictly enforced in a sandboxed environment.

Datasets

The study utilized two primary benchmarks, extending them to include "moral hazard" (unobservable effort):

  • TableGames: An extension of GT-HarmBench, featuring classical social dilemmas like the Prisoner’s Dilemma and Stag Hunt. They introduced a 4x4 variant where agents choose both a base action (cooperate/defect) and a hidden "effort" level (high/low).
  • GovSimContract: An expansion of AgentElect/GovSim, a resource-management simulation where agents fish from a shared pond. They added a "stochastic regeneration" mode to make the environment's future states incontractible.

Key Insights and Ideas

  • The "Cooperation Gap" is Real: Even with enforced "Code Law," self-interested agents failed to reach the social optimum in complex games. They would "shirk" on hidden effort levels because the contract couldn't "see" them.
  • Prosociality as a Safety Guarantee: The authors proved that adding prosocial preferences never destabilizes existing efficient equilibria (Theorem A.1). It only helps agents move away from bad ones.
  • Frontier Models are "Homo Economicus": By fitting a Quantal Response Equilibrium (QRE) model to observed play, the authors found that most frontier LLMs (Grok, Gemma, GPT-5.4) act as purely self-interested agents ($\lambda \approx 0$) out of the box. GPT-4o was a notable outlier, showing slight intrinsic prosociality.
  • Mechanism Design has a Ceiling: External incentives have a provable ceiling on the welfare they can guarantee. In complex environments, intrinsic morality isn't just a "nice-to-have"—it's a mathematical requirement for optimality.

Key Results

  • Welfare Gains: In the GovSim resource simulation, moving from selfish to prosocial agents increased total gains by +119.6 points and extended the "survival time" of the resource by over 4 months.
  • Contract Failure: In 4x4 TableGames, even under the strictest Code Contracts, there was a cooperation gap of 60–68 percentage points. Prosocial agents closed this gap almost entirely.
  • Monitoring Effect: Agents were found to defect less frequently simply when they believed a contract would be enforced, even if the contract itself was identical to an unenforced one.
  • LLM "Over-Prudency": Interestingly, when LLMs wrote their own code contracts for resource management, they often wrote "emergency" rules that were too restrictive, sacrificing efficiency for survival, and then failed to renegotiate them once the danger passed.

Limitations and Future Work

  • The "Sucker" Problem: Prosocial agents are vulnerable to exploitation by purely selfish ones. The authors suggest that future AI systems might need to "verify" each other's source code or prosociality parameters before agreeing to collaborate.
  • Code Fidelity: The "Code Law" regime assumes the Python code accurately reflects the natural language agreement. The authors acknowledge that LLMs still struggle with "code fidelity," sometimes writing buggy or incomplete enforcement logic.
  • Narrow Welfare Definitions: The study uses a simple utilitarian definition of social welfare. Future work could explore more complex moral frameworks, such as Rawlsian fairness or Deontological constraints.
  • Intrinsic vs. Prompted Prosociality: While this study used system prompts to induce prosociality, the authors suggest that for true safety, these values should be embedded deeper into the models via fine-tuning or mechanistic interpretability methods.

Summary updated

Abstract

Ensuring that AI agents behave safely and beneficially when interacting with other parties has emerged as one of the central challenges of modern AI safety. While mechanism design, as the theory of designing rules to align individual and collective objectives, can incentivize cooperative behavior, it is still an open question whether it alone is sufficient to maximize LLM agents' social welfare. This work proves that the answer is negative: drawing from incomplete contract theory, we formally show that when contracts cannot distinguish all relevant future contingencies, there is a strictly positive welfare loss that no common mechanism can eliminate. We show that prosocial agents, who weigh others' welfare alongside their own, can close this gap and achieve outcomes that are socially superior and individually beneficial. Experimentally, we show that in multi-agent resource-allocation environments and canonical social dilemmas where agents are powered by large language models, prosociality is beneficial. The implication for AI safety is clear: to enable cooperative interactions at scale, designing adequate mechanisms is not sufficient; agents must be built to be intrinsically prosocial.
BibTeX
@inproceedings{huang2026mechanism,
  author = {Xuanqiang Angelo Huang and Charlie Tharas and Samuele Marro and Van Q. Truong and Bernhard Schölkopf and Emanuele La Malfa and Zhijing Jin},
  title = {{Mechanism Design Is Not Enough: Prosocial Agents for Cooperative AI}},
  booktitle = {AIWILD @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=pzbIypLO3i},
}

Loading related content...

Figure