When self-interested agents operate in a shared environment, their actions inevitably spill over to affect one another. In economics, these unpriced side effects are known as externalities. A negative externality occurs when an agent's action imposes hidden costs on others (such as resource depletion or physical interference), while a positive externality occurs when an action yields uncompensated benefits (such as exploring an unknown resource or discovering valuable information).
Because standard reinforcement learning agents optimize solely for their private, immediate rewards, they remain blind to these spillover effects. This blind spot creates a "selfishness gap," frequently causing multi-agent systems to collapse into sub-optimal equilibria like the tragedy of the commons.
To bridge this gap, a growing body of research focuses on incentive alignment through reward internalization. Rather than forcing agents to be inherently altruistic, these frameworks modify the underlying reward landscapes so that external costs or benefits are directly reflected in an agent's private payoffs. By transforming the environment's economic structure, self-interested utility maximization naturally converges toward global social welfare.
Top-Down Alignment via Learned Pigovian Taxes
One classic economic solution to externalities is the Pigovian tax: a central authority levies charges on negative externalities and provides subsidies for positive ones. However, manually designing these tax rates in dynamic multi-agent reinforcement learning (MARL) environments is exceptionally difficult because agent preferences and environmental states shift continuously.
To automate this process, Yun Hua et al. in Learning Optimal "Pigovian Tax" in Sequential Social Dilemmas and LOPT: Learning Optimal Pigovian Tax in Sequential Social Dilemmas introduce a centralized "Tax Planner" agent. The planner's role is to observe the system, quantify the marginal social impact of each agent's behavior, and dynamically adjust individual reward functions.
[ Central Tax Planner ]
│ │
Subsidies (+) Taxes (-)
▼ ▼
[Agent A] [Agent B] ──► (Shared Environment)
By penalizing actions that harm the collective and subsidizing actions that benefit it, the planner reshapes the game matrix in real time. This mechanism ensures that individual agents find it individually rational to adopt policies that maximize collective social welfare.
A similar top-down approach can optimize performance within an individual agent's internal architecture. In Economic Hierarchical Q-Learning, Erik G. Schultink et al. apply this principle to hierarchical reinforcement learning. Here, parent agents act as localized planners, providing top-down financial subsidies to child agents. These subsidies allow child subtask agents to internalize the global mission consequences of their local actions, achieving hierarchical optimality without requiring heavy, centralized information sharing.
Peer-to-Peer Alignment via Coasean Bargaining
When a centralized planner is unavailable or impractical, agents can instead resolve externalities through direct negotiation. According to the Coase Theorem, if property rights are well-defined and transaction costs are low, self-interested parties can bargain to reach an efficient outcome regardless of the initial allocation of rights.
Antoine Scheid et al. operationalize this theorem for multi-agent bandit environments. In Learning to Mitigate Externalities: the Coase Theorem with Hindsight Rationality and Mitigating Externalities while Learning: an Online Version of the Coase Theorem, the authors demonstrate that even when agents are initially uncertain about their own utility functions and must learn them through trial and error, they can successfully negotiate optimal monetary transfers.
For instance, a downstream agent suffering from a negative externality can offer a calculated side payment to an upstream agent to compensate them for altering their action selection. This peer-to-peer transfer effectively alters the upstream agent's reward landscape, aligning its private incentives with the joint social optimum.
[Upstream Agent] ──(Action with Externality)──► [Downstream Agent]
▲ │
└─────────── [Monetary Transfer] ───────────────┘
This side-payment mechanism is equally effective for coordinating resource sharing. In Competitive Multi-armed Bandit Games for Resource Sharing, Hongbo Li et al. propose a coordination mechanism (CISP) that pairs aggregated data recommendations with monetary transfers. Agents who benefit from coordination make side payments to cover the costs incurred by other agents who explore less certain resources. This balances the system's budget while completely eliminating the efficiency gap caused by selfishness.
Endogenous Sharing and Market Stakes
Rather than relying on explicit taxes or formal bargaining rounds, agents can also internalize externalities by natively holding a financial stake in each other's success.
In Stochastic Market Games, Kyrill Schmid et al. introduce a framework where self-interested agents can trade "shares" of their future returns. If Agent A purchases a portion of Agent B's future rewards, Agent A's own profit becomes directly tied to Agent B's performance. Consequently, Agent A naturally internalizes the external costs of interfering with Agent B, which spontaneously incentivizes cooperative behaviors like mutual assistance and collision avoidance.
However, allowing agents to dynamically choose how much utility to share introduces new strategic vulnerabilities. Lenz Belzner et al. explore this dynamic in The Sharer's Dilemma in Collective Adaptive Systems of Self-Interested Agents using a mechanism called Distributed Optimization with Sharing (DOS). While agents can successfully optimize their sharing parameters to strategically steer the behavior of their peers toward globally efficient states, they also expose themselves to a "Sharer's Dilemma," where free-riding defectors can exploit overly generous agents.
Tying Rewards to Marginal Social Utility
A final avenue for internalization involves embedding the social value of an action directly into the market or auction mechanism itself.
This approach is highly effective in information aggregation systems, where agents often engage in wasteful "arms races" to report data first, even when hyper-speed provides zero marginal value to society. To eliminate this inefficiency, Grant Schoenebeck et al. design prediction markets in Timely Information from Prediction Markets that tie individual rewards directly to the marginal social utility of the information provided. Because the market structure forces agents to balance their private information-gathering costs against the actual societal value of timeliness, the system reaches a socially optimal equilibrium without requiring the market coordinator to know the agents' private cost functions or signal quality.
Similarly, auction mechanisms can be structured to account for informational spillovers. In Efficient Metadeliberation Auctions, David C. Parkes et al. demonstrate that when an agent's private research uncovers positive externalities that benefit third parties, the underlying auction remains efficient and incentive-compatible—provided that the uncovered social value is explicitly factored into the deliberating agent's payment and reward structure.
Key Foundational Work
The Sharer's Dilemma in Collective Adaptive Systems of Self-interested Agents (2018) — contributed 1 concept elements to this direction.
Efficient Metadeliberation Auctions (2008) — contributed 1 concept elements to this direction.
Economic Hierarchical Q-Learning (2008) — contributed 1 concept elements to this direction.