Learning Fairness in Multi-Agent Systems

Jiechuan Jiang, Zongqing Lu

This paper introduces FEN (Fair-Efficient Network), a hierarchical reinforcement learning framework designed to solve one of the thorniest problems in multi-agent systems: how to make agents both highly productive and fair to one another.

The Challenge: Efficiency vs. Fairness

In multi-agent reinforcement learning (MARL), agents typically maximize either their own local reward (leading to greed and inequality) or a shared global reward (which doesn't guarantee fairness). This often results in the Matthew Effect—the "rich get richer"—where more capable agents hog resources, leading to system instability and sub-optimal total performance.

The core challenge is that efficiency (total output) and fairness (equal distribution) are often conflicting objectives. Optimizing both simultaneously is a complex multi-objective problem that usually confuses a single-policy agent. Furthermore, most existing fairness solutions are handcrafted for specific tasks and require a centralized "god-view" of the system, making them hard to deploy in real-world, decentralized applications like traffic control or cloud computing.

The FEN Approach

The authors propose two primary innovations to tackle these challenges: a novel reward structure and a hierarchical architecture.

1. The Fair-Efficient Reward

Instead of relying on a global fairness metric, the authors derive a local reward for each agent. It uses the Coefficient of Variation (CV) of utilities to measure fairness. The reward is structured to encourage high average utility (efficiency) while strictly punishing agents whose utility deviates significantly from the group average (fairness).

2. Hierarchical Architecture

To resolve the conflict between being "greedy" for efficiency and "altruistic" for fairness, FEN utilizes a hierarchical model.

FEN Architecture
  • Sub-policies: The model features multiple low-level policies. One is dedicated to maximizing environmental rewards (efficiency). The others use an information-theoretic objective (maximizing mutual information) to learn diverse behaviors that might be needed to achieve fairness.
  • Controller: A high-level policy switches between these sub-policies every $T$ timesteps based on the current state of fairness in the system.

3. Fully Decentralized Training

To avoid the "curse of dimensionality," FEN uses a gossip algorithm. Agents estimate the global average utility by only communicating with their immediate neighbors, allowing the system to scale to many agents without a central server.

Evaluation Scenarios

The authors evaluated FEN on three distinct multi-agent "grid-world" style scenarios to test different aspects of cooperation and resource sharing:

Evaluation Scenarios
  • Job Scheduling (Left): 4 agents compete for a single resource that appears randomly. This tests basic resource sharing.
  • The Matthew Effect (Middle): 10 "Pac-man" agents eat ghosts. Eating a ghost makes an agent faster and larger, making it easier to eat more ghosts. This creates an inherent "rich-get-richer" trap.
  • Manufacturing Plant (Right): 5 heterogeneous agents require different types of "gems" to build products. This tests coordination, as the system's total output is limited by the agent that produces the least.

Key Results and Insights

FEN significantly outperformed baselines like Inequity Aversion and Maximin fairness across all tests:

  • Efficiency & Fairness: In Job Scheduling, FEN achieved 90% resource utilization with a fairness score (CV) of 0.17. For comparison, independent greedy agents had higher utilization (96%) but terrible fairness (CV of 1.57), while other fair baselines had much lower utilization.
  • Breaking the Matthew Effect: In the Pac-man scenario, FEN prevented a single agent from dominating. Interestingly, FEN achieved the highest social welfare (total ghosts eaten), proving that fairness can actually improve total system productivity by keeping all agents active and capable.
  • Coordination: In the Manufacturing Plant, FEN produced 48 products, more than double the next best baseline (23 products), because it successfully balanced the production rates of all five specialized agents.

Ultimately, the paper demonstrates that fairness can act as a powerful coordination mechanism. By separating conflicting goals into different sub-policies, the controller can make clean, high-level decisions, allowing the system to naturally converge toward Pareto efficiency and equal resource allocation.

Limitations and Future Work

  • Hyperparameter Tuning: The "fair-efficient" reward relies on constants ($c$ and $\epsilon$) that normalize the reward. Performance can be sensitive to these values.
  • Communication Overhead: While the gossip algorithm is decentralized, it still requires agents to exchange utility information with neighbors. In extremely bandwidth-constrained environments, this could be a bottleneck.
  • Homogeneous vs. Heterogeneous: While FEN worked in a manufacturing scenario with different agent roles, the "fairness" definition (equal utility) might need adjustment for systems where some agents are inherently more "expensive" or "important" than others.

Summary updated

Abstract

Fairness is essential for human society, contributing to stability and productivity. Similarly, fairness is also the key for many multi-agent systems. Taking fairness into multi-agent learning could help multi-agent systems become both efficient and stable. However, learning efficiency and fairness simultaneously is a complex, multi-objective, joint-policy optimization. To tackle these difficulties, we propose FEN, a novel hierarchical reinforcement learning model. We first decompose fairness for each agent and propose fair-efficient reward that each agent learns its own policy to optimize. To avoid multi-objective conflict, we design a hierarchy consisting of a controller and several sub-policies, where the controller maximizes the fair-efficient reward by switching among the sub-policies that provides diverse behaviors to interact with the environment. FEN can be trained in a fully decentralized way, making it easy to be deployed in real-world applications. Empirically, we show that FEN easily learns both fairness and efficiency and significantly outperforms baselines in a variety of multi-agent scenarios.
BibTeX
@inproceedings{jiang2019learning,
  author = {Jiechuan Jiang and Zongqing Lu},
  title = {{Learning Fairness in Multi-Agent Systems}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=vG0DTNXnthY},
}
Figure