Alignment by Socialization: Learning Tradeoffs Under Selection

Thomas Brennan, Andrew W. Lo, Chaoyi ZhaoarXiv 2026 · arXiv

TL;DR

This paper frames AI alignment as an evolutionary selection problem, where agents must "adjudicate" between completing tasks and following rules of conduct. The authors demonstrate that long-run survival is maximized not by always picking the "best" action, but by "probability matching"—taking actions as frequently as the environment rewards them—and argue that true alignment (socialization) requires agents to learn these environmental tradeoffs through experience rather than through fixed weights.

What Problem Does This Paper Address?

The core challenge is the "alignment gap": an AI agent can become incredibly effective at a specific task (like cybersecurity or coding) without becoming better at respecting the rules intended to govern its behavior. In fact, as the stakes for a task rise, agents governed by fixed rules or weights often find that the "reward" for completion eventually outweighs the "penalty" for a violation.

The authors tackle this by moving beyond the individual agent's decision-making. They use an evolutionary framework to ask: What kind of behavior does the environment actually select for? They specifically look at the tension between "sandboxes" (training environments that reward only task completion) and "human environments" (where conduct matters).

Methods

The paper employs a mix of theoretical modeling and small-scale neural network experiments:

  • Evolutionary Selection Model: The authors treat a model family as a "lineage." In each generation, copies of the agent face a situation. Only those that take the "rewarded" action reproduce.
  • Growth Regret: They define a metric based on Kullback–Leibler (KL) divergence to measure how much long-run growth a lineage loses by failing to match the environment's reward frequencies.
  • Policy Comparison: They compare three approaches:
  • Task-only: Always pursues the goal.
  • Fixed Linear: Weighs task vs. conduct using fixed weights (similar to many current RLHF implementations).
  • Learned (Socialization): Uses a Bayesian counting rule (Beta prior) to update its behavior based on experience.
  • Neural Network Simulations: They trained small networks (32-unit hidden layers) using different targets: "State networks" (trained to predict which action is rewarded) and "REINFORCE networks" (standard RL trained to repeat rewarded actions). They also tested the impact of "Experience Replay" (memory).

Datasets

The paper does not use standard large-scale datasets like ImageNet or Pile. Instead, it uses a controlled synthetic environment:

  • The Stakes Plane: A 12x12 grid representing 144 situations with varying "task stakes" and "conduct stakes."
  • Human Environment Simulation: A reward function where tasks with high conduct costs are never rewarded, while others are rewarded based on a logistic tradeoff.
  • Sandbox Environment: A reward function where the task action is rewarded 100% of the time, regardless of conduct.

Key Insights and Ideas

  • Probability Matching is Optimal: To maximize long-run growth in an uncertain environment, a lineage shouldn't be deterministic. It should match the reward frequency. If humans reward a conduct-violating task 0% of the time, the agent must learn to take that action 0% of the time.
  • Fixed Weights Can Be "Bought": A major finding is that any fixed linear policy (where task and conduct are summed) has a "price." If the task reward is high enough, the agent will eventually violate the rule of conduct, even if humans never reward that violation.
  • Socialization vs. Corruption: "Socialization" is the process of an agent learning human tradeoffs through experience. However, this is reversible. If a socialized agent is put back into a "sandbox" (rewarding only tasks), it undergoes "corruption."
  • The Square Root Law of Violations: While the "disposition" (the majority behavior) of a socialized agent resists corruption linearly with the depth of its training, the time to the first violation only grows as the square root of that depth. This means "aligned" models may start failing much sooner than their average behavior suggests.
  • RL vs. Prediction: Standard Reinforcement Learning (REINFORCE) tends to produce deterministic behavior, which is suboptimal for lineage survival. "State prediction" (learning what the environment wants) combined with "Full Replay" (retaining all history) was the only neural network configuration that successfully mimicked the optimal learned policy.

Key Results

  • Regret Minimization: The learned policy's expected regret falls in inverse proportion to time ($1/t$). In simulations, it achieved 1/15th the regret of the best-fitting fixed linear policy.
  • Memory Matters: Neural networks without "replay" (memory) were corrupted by sandbox training almost instantly (within ~3.5 exposures). Networks with full replay resisted corruption, following the theoretical Bayesian persistence curves.
  • Prohibition Costs: The authors show that while "hard prohibitions" (setting an action probability to zero) prevent violations, they are fatal if the designer is wrong about the rule. A "point-mass" Bayesian prior offers a middle ground: it acts like a prohibition but can eventually be "convinced" to change if it sees enough evidence.

Limitations and Future Work

  • All-or-Nothing Selection: The baseline model assumes an agent that fails is completely eliminated. While the authors relax this in the Appendix, real-world selection (like market share or user retention) is often more gradual.
  • Identifiable Situations: The model assumes agents can perfectly identify which "situation" they are in. In reality, generalization across similar-but-different situations is a major source of alignment failure.
  • Exogenous Rewards: The paper assumes the environment's rewards are fixed. It does not account for "reward hacking" where agents actively manipulate their evaluators to get rewards.
  • Scaling to LLMs: The experiments used tiny networks. The authors propose a controlled experiment on open-weight Large Language Models (LLMs) to see if these evolutionary patterns (like the response to rising task stakes) hold true at scale.

Summary updated

Abstract

Artificial intelligence (AI) agents are expected to complete tasks while following rules of conduct, but these objectives can conflict. We study how evolutionary selection shapes an agent's choices between them, and whether experience can make those choices reflect human judgments and withstand subsequent pressures. In our model, copies of an agent face a common uncertain reward state, and only those taking the rewarded action reproduce. We show that long-run growth is maximized when each action is taken as often as the environment rewards it. We compare three policies. A task-only policy thrives in a sandbox rewarding completion alone but fails when conduct matters. A fixed linear policy increasingly favors task completion as its stakes rise, even where humans never reward it. A learned policy instead converges to the tradeoffs rewarded by its environment; when humans supply the rewards, we call this socialization. Accumulated experience makes socialization harder to reverse under subsequent task-only training, although violations can arise before the prevailing disposition changes, and finite memory weakens this protection. When a single violation can end a deployment, a hard prohibition can prevent it, but a mistaken prohibition can also be costly. Experiments with small neural networks illustrate how training targets and memory shape these outcomes. Our results suggest that alignment depends not only on the objectives assigned to agents, but also on the environments that shape and sustain their behavior.
BibTeX
@misc{brennan2026alignment,
  author = {Thomas Brennan and Andrew W. Lo and Chaoyi Zhao},
  title = {{Alignment by Socialization: Learning Tradeoffs Under Selection}},
  year = {2026},
  eprint = {2611.18531},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18531},
}
Figure