Trading Safety Budgets to Escape Safe-but-Useless Equilibria

When several self-interested learning agents each operate under their own safety limit, they often settle into a dull outcome: nobody does the risky work that would make everyone better off, because each agent's own limit and own payoff make holding back the rational choice. The risky work might be clearing a hazardous area, entering a shared workspace, or scouting uncertain terrain. Existing fixes for social dilemmas let agents pay each other in reward, through gifts, transfer matrices, or enforceable contracts. Under hard per-agent safety limits, this cannot work in principle. Paying an agent more reward does not let it exceed its own cost budget. The cooperative outcome is not unattractive to that agent; it is infeasible for it.

We propose that agents should instead sign binding contracts that move safety budget itself. An agent that stays clear of the hazard hands part of its unused allowance of expected cost to the agent that goes in. The team's total cost stays exactly where it was, but the cooperative joint policy becomes feasible for the individual who has to carry it out. We claim that budget transfers, not reward transfers, are the missing instrument whenever per-agent constraints bind. We also claim they act against premature convergence: they keep constrained learners from locking into conservative, non-interactive policies early in training, before the cooperative mode is ever discovered.

Core Claim

A constrained Markov decision process maximizes return subject to a separate expected cost staying under a budget. Give each agent $i$ its own cost $C_i$ (expected discounted cost) and budget $\epsilon_i$. A budget contract specifies nonnegative transfers $\delta_{ij}$ from agent $i$ to agent $j$, contingent on observed behavior of $j$, for example that $j$ performs the hazardous public-good action. It is enforced by the environment, so each agent's constraint becomes

$$C_i(\pi) \le \epsilon_i - \sum_j \delta_{ij} + \sum_j \delta_{ji}, \qquad \epsilon_i - \sum_j \delta_{ij} + \sum_j \delta_{ji} \le \bar\epsilon,$$

Here $\bar\epsilon$ is a hard per-agent ceiling that no contract can breach. Transfers cancel in the sum, so total system cost is preserved by construction.

The theoretical part of the claim is simple and testable. Consider a joint policy that is socially optimal under the pooled budget but requires one agent to spend more than its own $\epsilon_i$. Such a policy cannot be made an equilibrium by any reward transfer. It can be made one by a budget transfer combined, if needed, with a reward transfer that compensates the donor.

The empirical part of the claim is that learners with only reward contracts fall into one of two failures. Either they converge to the conservative equilibrium, or, under soft Lagrangian enforcement, they reach cooperation only by accumulating violations. Learners with budget contracts reach cooperation within budget.

The ceiling matters for scope. Budget trading is appropriate when cost is fungible across agents, such as energy use, equipment wear, or aggregate risk exposure. It is not appropriate when the cost is harm to a specific agent that must never be concentrated. The ceiling $\bar\epsilon$ is how the proposal refuses to trade the latter.

Relation to Prior Work

Three lines of work on reward redistribution motivate the mechanism and supply its machinery. Resolving social dilemmas with minimal reward transfer computes, by linear programming, the smallest reward-transfer matrix that makes cooperation dominant. It shows that asymmetric sharing lets agents keep more of their own reward ($g^* = 0.75$ versus $s^* = 0.60$ in a 3-player cyclical Prisoner's Dilemma). Get It in Writing introduces MOCA, which pre-trains behaviors on a library of random contracts and then learns to propose and accept action-contingent reward contracts. It proves that rich enough contracts make the social optimum subgame-perfect. Emergent Prosociality Through Gifting shows that zero-sum reward gifts enlarge the basin of attraction of risky cooperative equilibria in Stag Hunt.

All three move only reward and assume actions carry no separate constrained cost. We reuse the LP formulation, extended with budget-transfer variables, and MOCA's two-phase training. We change what is transferred. MOCA's finding that unenforced gifting "frequently collapsed into selfish behavior" also constrains our design: transfers must be binding and environment-enforced.

Safe multi-agent reinforcement learning supplies the constrained optimizers and exposes the gap. Scalable Constrained Policy Optimization for Safe Multi-agent Reinforcement Learning (Scal-MAPPO-L) and DeCOM satisfy constraints for fully cooperative teams. They use fixed, pre-assigned budgets that cannot be pooled or traded, and they have no notion of individual incentive. Safe Multiagent Coordination via Entropic Exploration (E2C) is the closest work on the premature-convergence side. It avoids collapse into overly conservative team policies by rewarding novel observations rather than random actions, and reports collisions reduced by up to 50%. It assumes a shared team reward, so it cannot address agents who hold back because holding back is individually rational.

Safety Gymnasium documents the single-agent version of the failure we target: large penalties yield "safe but useless" policies. Policy Bifurcation in Safe Reinforcement Learning shows that single-mode policies get trapped when safe regions are disconnected. That is a representational cause of entrapment, distinct from the incentive cause studied here, and we control for it with an experiment. Goal Achievement Guided Exploration (GAGE) combats premature convergence by keeping a floor on action noise. This is exactly the kind of exploration increase that, per E2C, drives constrained agents across their cost boundary.

Research Program

The work has three parts.

Analysis. We extend the minimal-transfer LP of Willis et al. with budget-transfer variables and a per-agent ceiling. We characterize the class of constrained games in which reward transfers alone cannot support the pooled-budget optimum while budget transfers can. We compute the minimal budget transfer needed, in the spirit of their self-interest levels, for small constrained matrix games. This yields exact predictions for the learning experiments.

Learning system. We build a MOCA-style learner in which each agent's behavior policy is conditioned on its current effective budget. Each agent also has its own dual multiplier stabilized with a PID controller, as recommended in Safety Gymnasium, so one pre-trained policy can operate at whatever budget a contract assigns. Contracts are proposed and accepted as in MOCA. The contract space is extended from reward transfers to pairs of budget transfer and reward transfer, triggered by features of the counterparty's behavior. To avoid MOCA's single-proposer surplus capture muddying welfare comparisons, proposers rotate.

Measurement of entrapment. We track each agent's multiplier trajectory and the fraction of training time spent in the non-interactive mode. Following the gifting paper's basin-of-attraction framing, we also track the fraction of seeds that ever reach the cooperative mode. This lets us distinguish two ways contracts might help. They might help early, by keeping multipliers from driving agents into conservative policies before cooperation is found. Or they might help only after convergence.

Baselines

The naive reference is independent PPO-Lagrangian: each agent optimizes its own reward under its own fixed budget with no transfers. The relevant published figure is from Safety Gymnasium: in single-agent velocity tasks, PPO-Lag cut safety violations by 98% relative to PPO at a 45% loss of reward. We expect the multi-agent analogue to show the same conservatism in social-welfare terms. Beating it means higher social welfare at equal or lower cumulative cost.

The strongest existing competitor is MOCA-style reward contracting run on top of the same constrained learners. It is the best documented binding-redistribution method and is precisely what our claim says should fail when per-agent constraints bind. It will share our pre-training library, network sizes, number of environment steps, and hyperparameter search budget. The only difference is that its contracts cannot touch budgets. No reported result found on Lacuna for MOCA under cost constraints. The nearest reported number is gifting in high-risk Stag Hunt ($r=-10$), which raised cooperative success from 0% to 19.0% (Emergent Prosociality Through Gifting). In the unconstrained Stag Hunt setting, we treat that 19.0% as the bar our reward-contract reimplementation should reproduce before its constrained comparison is trusted. Beating MOCA means a significantly higher fraction of seeds reaching the cooperative mode with training-time violations at or below budget, which is the regime where reward contracts should fail.

The exploration competitor is E2C-style observation-novelty bonuses added to independent constrained learners with individual rewards. It is the strongest published method against conservative collapse in safe multi-agent reinforcement learning. No reported numeric return or cost on Safe MAMuJoCo was found on Lacuna for E2C or Scal-MAPPO-L. We will therefore rerun both in our tasks under matched budgets.

A centralized budget allocator jointly trained on the pooled constraint serves as an upper reference, not a competitor. It shows how much welfare decentralized contracting leaves on the table.

The key ablation compares budget-only contracts, reward-only contracts, and both. It isolates whether budget movement is the active ingredient. A second ablation removes the ceiling $\bar\epsilon$ to measure how much risk concentration the ceiling prevents and at what welfare cost.

Why not just…?

Why not just pay the risk-taker more reward? When the risk-taker's constraint binds, extra reward does not change what it is allowed to do, so the cooperative policy stays infeasible for it. This is the central claim and is directly falsifiable. If reward-only contracts match budget contracts at equal training-time violations in the constrained Cleanup and Stag Hunt variants, the claim is wrong. The honest risk is soft enforcement. Under Lagrangian methods constraints hold only asymptotically, so reward contracts may appear to work while quietly overspending. We therefore report cumulative training-time violations, not just final cost.

Why not just pool all budgets into one team constraint? A pooled constraint with individual rewards is a commons: each agent gains by spending shared budget, so pooling should make overspending by free riders worse, not better. A pooled budget with a team reward, as in E2C, abandons the self-interested setting altogether. We include the pooled self-interested variant as a baseline. If it matches budget contracts, the contracting machinery is unnecessary and we will say so.

Why not just explore more? If the conservative outcome is an equilibrium, better exploration finds cooperation but agents then abandon it, because it is not individually rational under their budgets. Evidence that raising action randomness hurts under strict constraints comes from Safe Multiagent Coordination via Entropic Exploration. We test observation-novelty exploration directly. The diagnostic is whether its seeds visit the cooperative mode and then leave it. That would indicate an incentive failure rather than an exploration failure.

Why not just add a big penalty instead of constraints? Safety Gymnasium shows that fixed penalties either tolerate catastrophic violations or produce safe-but-useless policies. GAGE's authors report that manual shaping creates new local optima. We include a swept penalty baseline for completeness but do not expect it to be competitive.

Proposed Experiments

We use three environments of increasing realism.

The first is a constrained Stag Hunt matrix game. Hunting stag incurs individual cost against a per-agent budget, and the pooled budget suffices for one hunter but not for each alone. Here the extended LP gives exact predictions, and the gifting benchmark's risk sweep provides the unconstrained reference.

The second is the Cleanup grid world used by MOCA, with a cost added for time spent in the polluted river. Cleaning is then both a public good and a budget drain.

The third is a Safe MAMuJoCo task built on the coupled robots of Scalable Constrained Policy Optimization for Safe Multi-agent Reinforcement Learning, with individual rewards and a hazard region. One agent's entry into the hazard raises everyone's return. To separate the incentive cause of entrapment from the representational one identified in Policy Bifurcation in Safe Reinforcement Learning, we run the continuous task with both single-Gaussian and mixture policies.

Metrics are social welfare, per-agent return (to detect exploitation of donors), final and cumulative training-time cost against each agent's effective budget, the maximum cost any single agent bears, and the fraction of seeds reaching the cooperative mode. We use at least 20 seeds per condition.

Three results would settle the proposal. First, in games where the extended LP says reward transfers are insufficient, budget contracts should reach cooperation within budget on a majority of seeds, while reward-only contracts either do not reach it or reach it only while violating constraints. Second, the benefit should appear early in training, visible as suppressed runaway growth of the risk-taker's multiplier, rather than only at convergence. Third, removing the ceiling should raise welfare only modestly while sharply raising the maximum per-agent cost. That result would show that the capped version buys most of the gain without concentrating harm.

If the first result fails, the budget-transfer thesis is false. If only the first holds, the mechanism is real but its link to premature convergence is not.