Abstract
Multi-agent reinforcement learning (MARL) often suffers from non-stationarity and the "tragedy of the commons," where agents converge on selfish behaviors that deplete shared resources. This paper investigates whether allowing agents to voluntarily gift rewards can stabilize learning. We compare "budgeted gifting" (cost-free) with "zero-sum gifting" (costly to the giver). Results in the Harvest environment demonstrate that zero-sum gifting effectively prevents resource collapse and improves collective returns, whereas budgeted gifting fails to incentivize cooperative behavior.
Introduction
Multi-Agent Reinforcement Learning (MARL) is notoriously difficult because the environment is constantly changing as agents learn and adapt to one another. This non-stationarity often leads to feedback loops where agents converge on selfish, destructive behaviors. A classic example is the Tragedy of the Commons. In scenarios where agents share a finite, regenerating resource, individual greed often leads to total resource depletion before the agents can learn to cooperate. While agents eventually learn to police each other to maintain sustainability, they typically undergo a disastrous period of total resource collapse first.
This research explores breaking the assumption that rewards are a fixed property of the environment. By allowing agents to pass rewards to one another, the study aims to stabilize the learning process and reach sustainable outcomes faster.
Methods
The hypothesis was tested in the Harvest environment, a grid-world where agents collect "apples" that regenerate based on the density of nearby apples. Over-harvesting leads to permanent resource depletion. Agents can also "tag" (temporarily remove) each other to reduce competition.
Two new action mechanics were introduced:
- Budgeted Gifting: Agents have a fixed "wallet" of reward units. Giving costs them nothing from their own score until the wallet is empty.
- Zero-Sum Gifting: Agents can gift rewards as often as they like, but every +1 given to a neighbor results in an immediate -1 penalty for the giver.
In both cases, the gift is distributed to agents in a small area directly in front of the gifting agent. Agents were trained using deep RL within a modified MiniGrid framework over 5,000 training episodes.
Results and Key Insights
The Failure of Budgeted Gifting
Surprisingly, giving agents a "free" budget to be altruistic did not improve outcomes. Because there was no individual incentive to use the gift action, agents ignored it, and the system succumbed to the tragedy of the commons, mirroring the baseline performance.
The Success of Zero-Sum Gifting
"Zero-sum" gifting—where the gift actually penalizes the giver—was the only mechanic that successfully stabilized the environment. It allowed agents to almost entirely avoid the resource depletion phase typically seen in early training.
As seen in Figure 1, average rewards per agent in the baseline and budgeted scenarios plummeted to near zero around episode 1,000 (the "tragedy" phase). With zero-sum gifting, this dip was nearly non-existent, maintaining a stable reward throughout training.
Social Dynamics and the Peace Metric
The data suggests that zero-sum gifting encourages agents to start "tagging" (punishing) greedy peers much earlier in the training process. This is reflected in the "Peace metric," which measures the effective population size.
Figure 2 shows that agents with zero-sum gifting reached a stable peace value much earlier than other groups. This indicates they learned to manage population density required for resource sustainability far more efficiently than agents in the other conditions.
Discussion and Future Work
It remains counter-intuitive that a penalized action (zero-sum gifting) leads to better outcomes than a free one (budgeted). The authors conjecture that the penalty might help shape the agent's policy by making the "gift" action more distinct during exploration, or that it fundamentally changes how agents value resource collection versus social interaction.
As a preliminary exploration, this work faces several limitations:
- Mechanism Clarity: Further theoretical work is needed to understand the interaction between the gift penalty and the agent's policy gradient.
- Environment Variety: Results are currently limited to the Harvest environment; testing on other social dilemmas like the Prisoner's Dilemma or Stag Hunt is necessary.
- Gifting Logic: Future work could explore more complex "trade" or "negotiation" mechanics where gifts are targeted or conditional rather than simple spatial broadcasts.
Summary updated