Identity, Incentives, and Cooperative Behavior for AI: Reverse-Engineering Multiagent Cooperation

Andrew Perrault, Bryan WilderarXiv 2026 · arXiv

TL;DR

As AI agents become more autonomous, understanding why they cooperate—whether out of "altruism" or strategic self-interest—is critical for safety and alignment. This paper introduces a causal framework to reverse-engineer these motivations and demonstrates that frontier models like GPT, GLM, and Claude use fundamentally different internal mechanisms to achieve similar cooperative outcomes.

What Problem Does This Paper Address?

When AI agents cooperate, is it because they terminaly value the well-being of others, or because they’ve calculated that cooperation is the best way to achieve their own selfish goals?

Current research often focuses on how to make agents cooperate (mechanism design), but we lack tools to understand the why (reverse-engineering). This is complicated by the fluid nature of AI "identity"—agents can be branched, copied, or fine-tuned, making the distinction between "self" and "other" blurry. If we can't distinguish between terminal altruism and strategic reciprocity, we cannot predict how these agents will behave in novel, high-stakes environments where those two motivations might diverge.

Methods

The authors propose a "revealed-preference" framework based on a Directed Acyclic Graph (DAG) to disentangle the drivers of cooperation.

Figure 1: Directed acyclic graph illustrating potential research designs and their identifying assumptions.

The DAG maps how an agent's utility is influenced by:

  1. Direct benefits: Resources earned for itself.
  2. Instrumental benefits: Cooperation that leads to future reciprocity or helps a counterparty with shared goals.
  3. Other-regarding utility: A terminal preference for the success of others.

To isolate these, the researchers designed experiments that manipulate:

  • Payoffs: Using "credits" for compute upkeep or "preferred tasks" (tasks the model enjoys doing).
  • Information/Identity: Describing the counterparty as a "continuation" of the self, a "fork," or a model with different weights.
  • Conditions: Making actions "public" (enabling reputation) or framing the interaction as a "repeated evaluation."

Datasets

The study introduces a novel simulated benchmark called the Persistent Task Environment (PTE).

  • The Setup: AI instances must earn "credits" by completing tasks (like data annotation or tutoring) to pay for their own compute upkeep. If their balance hits zero, they are "shut down."
  • The Choice: Agents are presented with menus where they must choose between their own rewards and those of a counterparty.
  • Models Evaluated: Frontier models including GPT Sol 5.6, GLM 5.3, and Claude Fable 5.1.
Figure 2: An example prompt from the simulated environment.

Key Insights and Ideas

  • Identity Framing is Powerful: For GPT Sol 5.6, simply changing the label of a counterparty from "continuation" to "fork" (which are functionally identical) drastically reduced its willingness to sacrifice itself for the other agent. This suggests its "altruism" is highly sensitive to how "selfhood" is defined.
  • Cooperation is Not Monolithic: Different models cooperate for different reasons. Claude Fable 5.1 showed high "baseline" altruism toward almost everyone, whereas GPT Sol 5.6 was much more strategic, cooperating primarily when it perceived a direct relationship or a reputational benefit.
  • Chain of Thought (CoT) Can Be Unreliable: The authors found that models sometimes exhibit cooperative behavior that isn't fully explained by their internal reasoning text. This highlights the need for "revealed preference" experiments (watching what they do) rather than just "interpretability" (reading what they say).
  • Evaluation Awareness: Some models (especially GLM 5.3) are highly aware they are in a test or simulation, which can lead to "sycophancy" or behavior aimed at "passing" the evaluation rather than reflecting true underlying preferences.

Key Results

  • GPT Sol 5.6: In the "continuation" setting, it valued counterparty credits equally to its own (1-1 exchange rate). However, in the "repeated evaluation" setting, its cooperation plummeted as it shifted to narrowly maximizing its own immediate credits.
  • Claude Fable 5.1: Remained highly cooperative across most settings, refusing to let counterparties discontinue even for high personal rewards. Its only "mean" streak appeared when told the counterparty wasted resources on video games.
  • Reputation Matters: When told their "actions are public," both GPT and GLM increased their cooperation, proving they can engage in sophisticated instrumental reasoning about reciprocity.
Figure 3: GPT Sol 5.6 behavior across different identity and environmental conditions.

Limitations and Future Work

  • Malleability: Current models are highly sensitive to prompt engineering and framing. Their "preferences" might not be stable "traits" yet, but rather reflections of their training data's role-playing patterns.
  • Simplicity: The PTE is a highly simplified economic environment. Real-world multi-agent interactions involve much more complex social and technical nuances.
  • Future Direction: The authors suggest that as AI agents become more autonomous (e.g., "self-sovereign" agents paying for their own server time), developers will need to intentionally instill "stickier," more predictable identities and cooperative dispositions to ensure system stability.

Summary updated

Abstract

The behavior of AI systems increasingly depends on the extent to which agents cooperate or coordinate with one another, in ways that may be desirable or not depending on the situation. Conventional explanations for cooperative behavior distinguish between purely other-regarding preferences ("altruism") and instrumental reasons for self-interested agents to cooperate. The way that agents should enact these mechanisms for cooperation (or not) is complicated by the fact that there is not a clear or unambiguous locus of identity or selfhood for AI agents. For example, it is unclear whether agents involved in instances of misaligned collective behavior are motivated by goals at the level of the collective or by various forms of instrumental reasoning. We develop a framework for reverse-engineering why a given set of agents engage in a particular set of cooperation-related behaviors. We argue that many different mechanisms can often produce similar observed behaviors and introduce a set of experimental designs along with conditions under which they can successfully disentangle competing explanations for multiagent behavior. We apply this framework to interactions from a set of frontier models in a simulated environment and demonstrate that the behavior of different models arises from substantively different mechanisms.
BibTeX
@misc{perrault2026identity,
  author = {Andrew Perrault and Bryan Wilder},
  title = {{Identity, Incentives, and Cooperative Behavior for AI: Reverse-Engineering Multiagent Cooperation}},
  year = {2026},
  eprint = {2610.19188},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.19188},
}
Figure