Evaluating Behavioral Alignment in Conflict Dialogue: A Multi-Dimensional Comparison of LLM Agents and Humans

Deuksin Kwon, Kaleen Shrestha, Bin Han, Elena Hayoung Lee, Gale LucasEMNLP 2025

TL;DR

This paper investigates how well Large Language Models (LLMs), when prompted with specific personalities, can mimic human behavior in adversarial conflict dialogues. By comparing LLM-generated dialogues with human-human dialogues across linguistic style, emotional expression, and strategic behavior, the study reveals that while GPT-4.1 shows strong alignment in language and emotion, significant gaps remain, particularly in strategic negotiation.

What Problem Does This Paper Address?

As LLMs become more sophisticated, they are increasingly used in complex, interactive tasks, including simulating human behavior in social scenarios. However, their ability to accurately replicate human nuances in emotionally charged and strategically complex situations, such as dispute resolution, remains largely unexplored. Dispute resolution requires not only strategic negotiation but also emotional sensitivity, making it a challenging benchmark for LLM alignment. This paper aims to bridge this gap by systematically comparing LLM agents with human participants in multi-turn, adversarial conflict dialogues, focusing on how well LLMs align with human behavior across multiple dimensions.

Methods

The researchers developed a multi-dimensional framework to evaluate behavioral alignment. This framework uses three key metrics:

  • Linguistic Style: Measured by the Linguistic Inquiry and Word Count (LIWC) features and Linguistic Entrainment (LE), which assesses how speakers adapt their language to their partner.
  • Emotional Dynamics: Analyzed through anger trajectories and magnitude, using Dynamic Time Warping (DTW) and Area Under the Curve (AUC) of anger intensity.
  • Strategic Behavior: Evaluated using the Interests-Rights-Power (IRP) framework, which categorizes negotiation strategies.

LLM agents were prompted with matched Five-Factor Model personality profiles to control for individual variation and enhance realism. These LLM dialogues were then compared to human-human dialogues from the KODIS dataset.

Datasets

The primary dataset used is the Kobe Dispute corpus (KODIS), a publicly available corpus of human-human dispute resolution dialogues. This dataset was chosen for its rich emotional and strategic annotations. The researchers also generated simulated LLM-LLM dialogues using the KODIS scenario and personality profiles.

Key Insights and Ideas

  • Personality Matters, But Isn't Enough: Prompting LLMs with specific personality profiles helps, but it doesn't fully bridge the gap to human-like behavior in complex social interactions.
  • GPT-4.1 Excels in Emotion and Language: GPT-4.1 demonstrated the closest alignment with human linguistic style and emotional dynamics, suggesting its strength in mimicking conversational flow and emotional expression.
  • Claude-3.7-Sonnet Shines in Strategy: Claude-3.7-Sonnet, however, showed better alignment with human strategic negotiation behavior, indicating its capability in tactical decision-making within disputes.
  • Anger Dynamics are Complex: While LLMs can mimic anger trajectories, human anger exhibits greater variance and variability, highlighting a subtle but important difference in emotional expression.
  • Strategic Behavior is More Than Words: The study suggests that strategic negotiation patterns (IRP) are not always fully captured by surface-level linguistic cues, underscoring the distinction between language use and underlying strategic reasoning.

Key Results

  • GPT-4.1 showed the closest overall behavioral alignment with humans, particularly in linguistic style and emotional dynamics.
  • Claude-3.7-Sonnet demonstrated the best alignment in strategic negotiation behavior.
  • LLMs, in general, tended to negotiate longer than humans.
  • While LLMs can replicate anger trajectories, human anger intensity shows broader variance.
  • Claude's IRP distribution closely aligned with human patterns, indicating strong convergence in decision-making.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited LLM Range: Resource constraints prevented testing a wider array of LLMs, limiting the scope of comparison.
  • Dataset Realism: The KODIS dataset, while rich, is a role-play setup and may not perfectly reflect real-world negotiation dynamics.
  • Inconsistencies in LLMs: Some LLMs showed inconsistencies between their stated issue importance and their actual negotiation behavior, suggesting a potential lack of robust strategic reasoning.

Future work could involve:

  • Testing a broader range of open-source LLMs.
  • Developing improved prompting strategies or alternative methods to enhance LLMs' strategic decision-making for more realistic simulations.
  • Exploring more nuanced modeling of strategic reasoning and emotional dynamics to further improve LLM alignment in complex social interactions.

Summary updated

BibTeX
@inproceedings{kwon2025evaluating,
  author = {Deuksin Kwon and Kaleen Shrestha and Bin Han and Elena Hayoung Lee and Gale Lucas},
  title = {{Evaluating Behavioral Alignment in Conflict Dialogue: A Multi-Dimensional Comparison of LLM Agents and Humans}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure