Abstract
Real-world multi-agent scenarios often involve mixed motives, demanding altruistic agents capable of self-protection against potential exploitation. However, existing approaches often struggle to achieve both objectives. In this paper, based on that empathic responses are modulated by inferred social relationships between agents, we propose LASE (Learning to balance Altruism and Self-interest based on Empathy), a distributed multi-agent reinforcement learning algorithm that fosters altruistic cooperation through gifting while avoiding exploitation by other agents in mixed-motive games. LASE allocates a portion of its rewards to co-players as gifts, with this allocation adapting dynamically based on the social relationship—a metric evaluating the friendliness of co-players estimated by counterfactual reasoning. Comprehensive experiments demonstrate LASE's ability to promote group collaboration without compromising fairness and its capacity to adapt policies to various types of interactive co-players.
Introduction
Multi-agent systems typically oscillate between two extremes: pure cooperation or pure competition. However, most real-world scenarios—from autonomous driving to international trade—are "mixed-motive." In these environments, agents face a critical dilemma: excessive selfishness leads to sub-optimal Nash Equilibria (a "race to the bottom"), while excessive altruism invites exploitation by "free-riders." Existing solutions often rely on centralized controllers, which are impractical for independent agents. The core challenge lies in enabling a decentralized agent to decide when and how much to assist others based solely on local observations.
The LASE Framework
LASE introduces a computational model of empathy that modulates an agent's response based on inferred social relationships. The architecture consists of three integrated components:
- Social Relationship Inference (SRI): The core module uses counterfactual reasoning to evaluate co-players. It compares the actual reward received from a joint action against a counterfactual baseline, determining how a co-player's specific action influenced the agent's outcome relative to their typical behavior.
- Perspective-Taking (PT): To overcome the limitations of decentralized observations, the PT module employs an observation conversion network to simulate a co-player's perspective and predict their policy.
- Zero-Sum Gifting: Agents dynamically "gift" a portion of their reward to others based on the SRI output. This zero-sum mechanism ensures that the agent's altruism is an investment, creating a direct incentive for co-players to behave cooperatively.
Experimental Evaluation
The researchers evaluated LASE across several Sequential Social Dilemmas (SSDs) that test different facets of cooperation and coordination:
- Iterated Prisoner’s Dilemma (IPD): The foundational matrix game for testing cooperation vs. defection.
- Coin Game: Agents must collect coins of their own color; collecting a rival's color penalizes the rival.
- Cleanup: A public goods game requiring agents to perform unrewarded "dirty work" (cleaning a river) to enable collective benefits (apple growth).
- Sequential Stag-Hunt (SSH): A coordination game balancing high-reward teamwork against safe solo tasks.
- Sequential Snowdrift Game (SSG): A game focused on the distribution of costly tasks that benefit the group.
Key Results and Insights
LASE demonstrates significant improvements over standard agents and high-profile baselines like LOLA and LIO:
- Cooperation Stability: In the Prisoner's Dilemma, while standard A2C agents converge to 100% defection, LASE agents achieve cooperation with a ~93% probability.
- Solving the "Dirty Work" Problem: In Cleanup, LASE enables the emergence of a sustainable economy where "collectors" reward "cleaners" through gifts, making individual sacrifice viable.
- Emergent Fairness: LASE promotes group collaboration without compromising fairness. It distributes rewards more equitably than "Group Optimal" strategies, which often maximize total utility at the expense of individual agents.
- Adaptability: LASE successfully identifies and rewards cooperative behaviors while withholding gifts from defectors, effectively protecting itself from exploitation.
Conclusion and Future Work
LASE provides a robust framework for balancing altruism and self-interest through empathic modeling. Future research will address current limitations, including the possibility of gift refusal by agents, extending the framework to continuous action spaces (e.g., robotics), and improving inference robustness in highly noisy or partially observable environments.
Summary updated