Learning Fairness in Multi-Agent Systems

Jiechuan Jiang, Zongqing Lu

Abstract

Fairness is essential for human society, contributing to stability and productivity. Similarly, fairness is also the key for many multi-agent systems. Taking fairness into multi-agent learning could help multi-agent systems become both efficient and stable. However, learning efficiency and fairness simultaneously is a complex, multi-objective, joint-policy optimization. To tackle these difficulties, we propose FEN, a novel hierarchical reinforcement learning model. We first decompose fairness for each agent and propose fair-efficient reward that each agent learns its own policy to optimize. To avoid multi-objective conflict, we design a hierarchy consisting of a controller and several sub-policies, where the controller maximizes the fair-efficient reward by switching among the sub-policies that provides diverse behaviors to interact with the environment. FEN can be trained in a fully decentralized way, making it easy to be deployed in real-world applications. Empirically, we show that FEN easily learns both fairness and efficiency and significantly outperforms baselines in a variety of multi-agent scenarios.

Introduction

In human society, stability is predicated on a balance between productivity and fairness. In Multi-Agent Reinforcement Learning (MARL), however, the traditional focus has been on maximizing "social welfare"—the aggregate sum of rewards. This often results in "winner-take-all" dynamics, where a subset of agents monopolizes resources, leading to system instability and poor long-term outcomes. To address this, we introduce the Fair-Efficient Network (FEN), a hierarchical model designed to integrate fairness directly into the learning process without compromising overall efficiency.

The Challenge of Fairness in MARL

Learning fairness in a multi-agent setting presents three primary hurdles:

  1. Multi-objective Conflict: Efficiency (maximizing total utility) and fairness (equitable distribution) are often competing goals. A monolithic policy network often fails to optimize both simultaneously.
  2. Global vs. Local Information: Fairness is inherently a global property, yet agents in decentralized systems typically only have access to local observations.
  3. Generalization: Many existing fair RL solutions are domain-specific and lack the flexibility to generalize across different tasks.

The Fair-Efficient Network (FEN)

FEN addresses these challenges through a decentralized, hierarchical architecture that decomposes global fairness into individual incentives.

1. Fair-Efficient Reward

Rather than a global objective, FEN provides each agent $i$ with a local reward $r_t^i$ that balances efficiency and fairness. The reward encourages agents to increase the group's average utility ($\bar{u}$) while penalizing deviations of their own utility ($u^i$) from that average. This structure incentivizes agents to reach the Pareto frontier while minimizing variance across the population.

2. Hierarchical Architecture

To manage the inherent tension between efficiency and fairness, FEN utilizes a two-level hierarchy.

The FEN hierarchical architecture featuring a controller and multiple sub-policies.
  • The Controller: A high-level policy $\theta$ that receives the fair-efficient reward. It selects which low-level sub-policy to activate for a fixed duration $T$.
  • Sub-policies: A set of low-level policies $\phi_1, \dots, \phi_M$ that interact with the environment. One sub-policy typically focuses on raw efficiency, while others are trained using information-theoretic objectives (maximizing mutual information) to ensure a diverse set of behaviors is available to the controller.

3. Decentralized Consensus

To maintain decentralization, FEN employs a gossip algorithm. Agents share their local utility estimates with neighbors, allowing the entire system to converge on an accurate estimate of the global average utility without requiring a central server.

Experimental Evaluation

We evaluated FEN across three distinct environments designed to test different aspects of fairness and coordination.

The three experimental environments: Job Scheduling (left), The Matthew Effect (center), and Manufacturing Plant (right).
  1. Job Scheduling: Agents must share a single moving resource in a grid.
  2. The Matthew Effect: A "rich-get-richer" scenario where consuming "ghosts" increases an agent's speed and size, facilitating further consumption.
  3. Manufacturing Plant: A complex task where agents collect different gems to assemble products. Total output is constrained by the agent producing the least (the bottleneck), making fairness a functional necessity for efficiency.

Key Results and Insights

FEN significantly outperformed standard baselines, including Independent PPO and Maximin strategies, across all metrics:

  • Efficiency vs. Fairness: In Job Scheduling, FEN achieved 90% resource utilization with a low Coefficient of Variation (CV) of 0.17. While independent agents were slightly more efficient (96%), they were extremely unfair (CV of 1.57), leaving some agents with zero resources.
  • Breaking Monopolies: In the Matthew Effect, FEN successfully prevented the "rich-get-richer" cycle, achieving high social welfare with near-perfect fairness (CV of 0.06).
  • Unlocking Performance through Fairness: In the Manufacturing Plant, FEN produced 48 products—more than double the next best baseline (23 products). This demonstrates that in interdependent systems, fairness is the key to maximizing total productivity.

Conclusion

FEN demonstrates that hierarchy can act as an effective "conflict resolver" in multi-objective MARL. By switching between specialized sub-policies, the controller can balance selfish efficiency with prosocial fairness. Future work will explore FEN's performance in heterogeneous systems and its scalability to significantly larger agent populations.

Reference: Jiang, J., & Lu, Z. (2019). Learning Fairness in Multi-Agent Systems. 33rd Conference on Neural Information Processing Systems (NeurIPS 2019).

Summary updated

Abstract

Fairness is essential for human society, contributing to stability and productivity. Similarly, fairness is also the key for many multi-agent systems. Taking fairness into multi-agent learning could help multi-agent systems become both efficient and stable. However, learning efficiency and fairness simultaneously is a complex, multi-objective, joint-policy optimization. To tackle these difficulties, we propose FEN, a novel hierarchical reinforcement learning model. We first decompose fairness for each agent and propose fair-efficient reward that each agent learns its own policy to optimize. To avoid multi-objective conflict, we design a hierarchy consisting of a controller and several sub-policies, where the controller maximizes the fair-efficient reward by switching among the sub-policies that provides diverse behaviors to interact with the environment. FEN can be trained in a fully decentralized way, making it easy to be deployed in real-world applications. Empirically, we show that FEN easily learns both fairness and efficiency and significantly outperforms baselines in a variety of multi-agent scenarios.
BibTeX
@inproceedings{jiang2019learning,
  author = {Jiechuan Jiang and Zongqing Lu},
  title = {{Learning Fairness in Multi-Agent Systems}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=vG0DTNXnthY},
}
Figure