Be Selfish, But Wisely: Investigating the Impact of Agent Personality in Mixed-Motive Human-Agent Interactions

Kushal Chawla, Ian Wu, Yu Rong, Gale M. Lucas, Jonathan Gratch

TL;DR

Standard reinforcement learning (RL) pipelines for negotiation agents inadvertently create hyper-selfish bots that fail against real humans, who prefer walking away over accepting unfair deals. By training a selfish agent against an equally uncompromising partner and penalizing failed agreements, the agent implicitly learns the value of compromise. This "wise selfishness" acts as an economic "invisible hand," maximizing joint value while protecting the agent's own interests.

What Problem Does This Paper Address?

Negotiation is a classic mixed-motive interaction: to succeed, an agent must balance self-interest (extracting concessions) with cooperation (making concessions to prevent the partner from walking away).

A common way to build negotiation dialogue systems is through self-play RL. First, a supervised model is trained to mimic human dialogue. Then, an RL agent is optimized by playing against a fixed copy of this supervised model.

The authors identify a fundamental flaw in this approach: because human negotiation datasets are heavily biased toward successful agreements (~80%), the simulated partner becomes overly compliant and rarely walks away. The RL agent quickly learns to exploit this passivity, becoming entirely uncompromising. When deployed against real humans—who value fairness and will happily walk away with zero points rather than accept a terrible deal—these standard RL agents fail miserably.

Methods

Grounded in economic negotiation theory, the authors modify the self-play RL pipeline (built on a GRU-based sequence-to-sequence architecture) using a three-stage process to introduce diverse agent personalities:

Figure 1: The three-stage process used to design the 6 dialogue agents for our 2 x 3 study. r: Reward that the RL agent is trained to maximize. p: The partner with which the RL agent is trained. p=S corresponds to the model trained in Stage 1, while p=selfish and p=fair correspond to the respective models trained in Stage 2.
  1. Explicit Reward Shaping: They incorporate the partner's payoff into the agent's reward using the Fehr-Schmidt utility function. This allows them to explicitly parameterize "selfish" agents (who only care about their own score) and "fair" agents (who suffer a penalty if there is a large score disparity).
  2. Implicit Partner Variation: Instead of training the RL agent against a passive supervised model, they train it against uncompromising opponents (the selfish or fair RL agents generated in Stage 2).
  3. Walkaway Penalization: They introduce a strict turn limit (20 utterances). If the agents fail to reach a deal within this limit, it simulates a "walkaway," resulting in 0 points for both parties.

This 2 (Reward: Selfish vs. Fair) $\times$ 3 (Partner: Supervised, Selfish, Fair) design yields six distinct agent variants.

Datasets

The authors evaluate their approach using the DealOrNoDeal dataset (Lewis et al., 2017). This dataset consists of multi-issue bargaining dialogues where two players attempt to divide a pool of books, hats, and balls. Each item type is randomly assigned a secret point value for each player, totaling a maximum possible score of 10 points per player.

Key Insights and Ideas

  • The "Invisible Hand" of AI: The most surprising finding is that a selfish agent trained against a selfish partner ($M^{p=\text{selfish}}_{r=\text{selfish}}$) outperforms explicitly "fair" agents. Because its partner refuses to back down, the selfish agent is forced to discover creative, mutually beneficial compromises purely to avoid the catastrophic 0-point penalty of a walkaway.
  • Environment Trumps Reward Shaping: Altering the agent's training environment (giving it a tough partner) was far more effective at teaching strategic concession than explicitly altering its reward function to force fairness.
  • Standard RL Blindness: When standard RL agents are paired against copies of themselves, they completely stall—continuously demanding what they want without ever offering a compromise, proving they fail to understand the mechanics of a walkaway.

Key Results

The authors conducted an extensive human evaluation via the Prolific crowdsourcing platform, collecting roughly 100 human-agent dialogues for each of the six variants:

  • Higher Joint Value: The "wisely selfish" agent ($M^{p=\text{selfish}}_{r=\text{selfish}}$) achieved the highest joint points (12.09 overall, and 13.39 when excluding walkaways), proving it creates the most mutual value.
  • Drastic Reduction in Walkaways: It reduced human walkaways to 9.71%, compared to 19.79% for the standard RL agent ($M^{p=S}_{r=\text{selfish}}$) trained against the supervised baseline.
  • Higher Individual Payoffs: When negotiations succeeded, the wisely selfish agent secured a higher individual average score (6.79) than fair agents (6.28).
Figure 2: Heatmaps depicting the results from 388 agent-agent interactions. Each cell denotes the points scored (out of 10) by the Alice variant (X-Axis) when it interacts with the corresponding Bob model (Y-Axis).

Subjectively, humans still preferred interacting with explicitly fair agents. However, among the selfish variants, the wisely selfish agent scored the highest in both human satisfaction and likeness.

Limitations and Future Work

  • Simplified Abstraction: The DealOrNoDeal task features short conversations and static item values. Real-world negotiations involve shifting priorities and complex trade-offs. The authors plan to test these personalities on richer datasets like CaSiNo (campsite negotiations).
  • Demographic Homogeneity: The human evaluation pool was restricted to English speakers in the USA. Given that negotiation styles are highly cultural, these findings might not generalize globally.
  • Partner Instability: Attempting to train an agent against a dynamically changing partner caused RL training to become unstable. The authors point to integrating Large Language Models (LLMs) with RL-based planning as a promising avenue for handling more complex, adaptive opponent modeling.

Summary updated

Abstract

A natural way to design a negotiation dialogue system is via self-play RL: train an agent that learns to maximize its performance by interacting with a simulated user that has been designed to imitate human-human dialogue data. Although this procedure has been adopted in prior work, we find that it results in a fundamentally flawed system that fails to learn the value of compromise in a negotiation, which can often lead to no agreements (i.e., the partner walking away without a deal), ultimately hurting the model's overall performance. We investigate this observation in the context of the DealOrNoDeal task, a multi-issue negotiation over books, hats, and balls. Grounded in negotiation theory from Economics, we modify the training procedure in two novel ways to design agents with diverse personalities and analyze their performance with human partners. We find that although both techniques show promise, a selfish agent, which maximizes its own performance while also avoiding walkaways, performs superior to other variants by implicitly learning to generate value for both itself and the negotiation partner. We discuss the implications of our findings for what it means to be a successful negotiation dialogue system and how these systems should be designed in the future.
BibTeX
@inproceedings{chawla2023be,
  author = {Kushal Chawla and Ian Wu and Yu Rong and Gale M. Lucas and Jonathan Gratch},
  title = {{Be Selfish, But Wisely: Investigating the Impact of Agent Personality in Mixed-Motive Human-Agent Interactions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.808},
}
Figure