Be Selfish, But Wisely: Investigating the Impact of Agent Personality in Mixed-Motive Human-Agent Interactions

Kushal Chawla, Ian Wu, Yu Rong, Gale Lucas, Jonathan Gratch

Abstract

A natural way to design a negotiation dialogue system is via self-play RL: train an agent that learns to maximize its performance by interacting with a simulated user that has been designed to imitate human-human dialogue data. Although this procedure has been adopted in prior work, we find that it results in a fundamentally flawed system that fails to learn the value of compromise in a negotiation, which can often lead to no agreements (i.e., the partner walking away without a deal), ultimately hurting the model's overall performance. We investigate this observation in the context of the DealOrNoDeal task, a multi-issue negotiation over books, hats, and balls. Grounded in negotiation theory from Economics, we modify the training procedure in two novel ways to design agents with diverse personalities and analyze their performance with human partners. We find that although both techniques show promise, a selfish agent, which maximizes its own performance while also avoiding walkaways, performs superior to other variants by implicitly learning to generate value for both itself and the negotiation partner. We discuss the implications of our findings for what it means to be a successful negotiation dialogue system and how these systems should be designed in the future.

Introduction

Negotiation is a classic "mixed-motive" interaction where participants must balance self-interest with the cooperation necessary to reach an agreement. In automated systems, a common failure mode is the emergence of "greedy" agents. These agents, often trained via reinforcement learning (RL) against simulated partners that mimic cooperative human behavior, learn to exploit their partners. However, when faced with real humans who may walk away from unfair deals, these greedy agents fail to reach any agreement, resulting in zero points for both parties.

This research identifies a fundamental flaw in standard self-play RL: the lack of "no-deal" or "walkaway" scenarios in training data. Because human datasets like DealOrNoDeal are biased toward successful agreements, simulated partners rarely walk away, teaching RL agents that they can be uncompromising bullies without consequence.

Methodology: Designing Diverse Personalities

To address this, the researchers introduced a three-stage training pipeline grounded in economic theory, specifically the Fehr and Schmidt utility function. This allows for the creation of agents with distinct "personalities" based on their reward structures and the partners they train against.

Figure 1: The three-stage training pipeline. Stage 1 uses supervised learning. Stage 2 introduces RL with varying reward functions (Fair vs. Selfish). Stage 3 varies both the reward function and the training partner's personality.

The methodology focuses on two primary levers:

  1. Explicit Reward Variation: Agents are rewarded based on their own points (Selfish) or a combination that penalizes inequality (Fair).
  2. Implicit Partner Variation: Instead of training against a "soft" supervised bot, agents are trained against uncompromising, selfish partners.

A critical addition was a hard turn limit (20 utterances). If no agreement is reached within this limit, it is recorded as a "walkaway," and both agents receive zero reward. This forces the agent to consider the risk of total failure. The standout model variant is the "Selfish-Wise" agent ($M^{p=selfish}_{r=selfish}$), which uses a selfish reward function but is trained against a selfish partner.

Key Insights and Results

The core finding is that the most successful agent is not the one explicitly programmed to be "fair," but the one that learns to be "wisely selfish."

The "Invisible Hand" of AI

The "Selfish-Wise" agent implicitly learns the value of compromise. By training against a tough partner, it discovers that yielding slightly is a strategic necessity to prevent a walkaway and secure its own reward. This behavior mirrors the "invisible hand" concept, where self-interested actions lead to mutually beneficial outcomes.

Performance Metrics

In human trials involving nearly 600 conversations on the Prolific platform, the "Selfish-Wise" agent demonstrated superior performance:

  • Joint Value: It achieved the highest joint points (12.09), outperforming "fair" variants.
  • Agreement Rate: While other selfish models suffered from walkaway rates as high as 24%, the "Selfish-Wise" agent reduced this to 9.7%.
  • Human Benefit: Human partners secured more points (6.18) against this agent than any other, indicating a more balanced and productive negotiation.
Figure 6: Mean joint points (excluding disagreements) across different model pairings. The Selfish-Wise agent ($M^{p=selfish}_{r=selfish}$) consistently contributes to high joint value in self-play and cross-play scenarios.

Conclusion

Building robust negotiation agents requires exposing them to the possibility of failure. Training against "yes-man" simulators creates fragile models that cannot handle real-world friction. By incorporating "walkaways" as essential training data and varying partner personalities, we can develop agents that understand the strategic value of compromise. Future work will explore integrating these RL strategies with Large Language Models to enhance reasoning and natural language persuasion in more complex, high-stakes negotiation scenarios.

Summary updated

Abstract

A natural way to design a negotiation dialogue system is via self-play RL: train an agent that learns to maximize its performance by interacting with a simulated user that has been designed to imitate human-human dialogue data. Although this procedure has been adopted in prior work, we find that it results in a fundamentally flawed system that fails to learn the value of compromise in a negotiation, which can often lead to no agreements (i.e., the partner walking away without a deal), ultimately hurting the model's overall performance. We investigate this observation in the context of the DealOrNoDeal task, a multi-issue negotiation over books, hats, and balls. Grounded in negotiation theory from Economics, we modify the training procedure in two novel ways to design agents with diverse personalities and analyze their performance with human partners. We find that although both techniques show promise, a selfish agent, which maximizes its own performance while also avoiding walkaways, performs superior to other variants by implicitly learning to generate value for both itself and the negotiation partner. We discuss the implications of our findings for what it means to be a successful negotiation dialogue system and how these systems should be designed in the future.
BibTeX
@inproceedings{chawla2023be,
  author = {Kushal Chawla and Ian Wu and Yu Rong and Gale Lucas and Jonathan Gratch},
  title = {{Be Selfish, But Wisely: Investigating the Impact of Agent Personality in Mixed-Motive Human-Agent Interactions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.808},
}
Figure