Evaluating Conversational Agents with Persona-driven User Simulations based on Large Language Models: A Sales Bot Case Study

Justyna Gromada, Alicja Kasicka, Ewa Komkowska, Lukasz Krajewski, Natalia Krawczyk, Morgan Veyret, Bartosz Przybył, Lina M. Rojas-Barahona, Michał K. SzczerbakEMNLP 2025

TL;DR

This paper introduces a novel method for evaluating conversational AI agents, specifically a sales bot, by using Large Language Models (LLMs) to simulate diverse customer personas. This approach offers a scalable and cost-effective alternative to human evaluation, demonstrating that LLM-based simulators can effectively mimic nuanced customer behaviors and that cross-selling strategies can be implemented without significantly impacting customer satisfaction.

What Problem Does This Paper Address?

Evaluating conversational agents (CAs) before deployment is crucial for ensuring functionality, performance, and identifying weaknesses. While human evaluation is the gold standard, it's expensive and time-consuming. Existing user simulation approaches offer scalability but often struggle to capture the complexity and nuance of real human interactions. This paper addresses the challenge of creating realistic, persona-driven user simulations for evaluating CAs, particularly in a sales context, where understanding customer behavior, assertiveness, and precision of needs is critical for effective interaction and business outcomes.

Methods

The core of this work is a Persona-driven User Simulator built using LLMs. The methodology involves three key steps:

  1. Persona Generation: LLMs (specifically Anthropic Claude 3.5 Sonnet v2) were used to generate 100 diverse customer personas representing a real customer base, incorporating demographic and behavioral traits.
  2. Goal and Scenario Definition: For each persona, specific conversation goals (e.g., phone upgrade, replacement) and four conversation scenarios (varying assertiveness and precision of needs) were created.
  3. LLM-based Simulation: These personas, goals, and scenarios were fed into an LLM-based user simulator (Google Gemini 1.5-flash) to interact with the sales bot. This simulator was configured to adopt the persona's traits and follow the defined goals and scenarios.

The evaluation framework itself employed both human annotators and various LLMs (commercial and open-source) to assess:

  • User Simulator Quality: Persona, goal, and scenario compliance.
  • SALESBOT 2.0 Performance: User satisfaction, helpfulness, naturalness, and proactiveness.

Datasets

The paper introduces a newly created dataset of 100 customer personas, along with corresponding goals and conversation scenarios. This dataset is designed for evaluating conversational agents in a sales context and is publicly available on Hugging Face. The dataset was used to generate 5200 unique dialogues for evaluation.

Key Insights and Ideas

  • LLMs as Realistic Simulators: LLMs can effectively generate and embody diverse customer personas, mimicking nuanced traits like assertiveness and precision of needs, which is crucial for realistic evaluation.
  • Scalability of LLM Simulation: The LLM-based simulation approach offers a significant cost and time advantage over human evaluation, making it feasible to test a large number of configurations and scenarios.
  • User Satisfaction as a Reliable Metric: User satisfaction emerged as the most reliably assessed dimension across both human and LM evaluators, suggesting it's a robust indicator of CA performance.
  • Persona-Driven Sales Strategies: The study highlights how different customer personas (e.g., less assertive vs. highly assertive, low vs. high precision of needs) respond differently to sales strategies, particularly cross-selling.
  • Cross-selling Effectiveness: Implementing cross-selling strategies had a minimal impact on overall customer satisfaction, demonstrating that business objectives can be pursued without alienating customers, provided the approach is tailored.

Key Results

  • User Simulation Quality: Human evaluation showed high scores for persona (4.89/5), goal (4.76/5), and scenario (4.45/5) compliance. LLM evaluations, particularly by DeepSeek-R1-Distill-Qwen-14B, demonstrated strong alignment with human ratings, achieving over 99.7% of dialogues rated ≥4.
  • SALESBOT 2.0 Performance: User satisfaction was the most consistently high-scoring metric across evaluators. Less assertive customers with low precision of needs reported higher satisfaction than highly assertive customers with high precision of needs.
  • Cross-selling Impact: Across various cross-selling configurations, user satisfaction remained largely stable compared to the baseline, with no significant negative differences observed.
  • LM Evaluator Performance: Commercial models like GPT-40 and Claude 3.5 Sonnet v2, and open-source models like DeepSeek-R1-Distill-Qwen-14B, showed strong performance in evaluating both simulation quality and bot performance, often at a lower cost than human evaluation.

Limitations and Future Work

Limitations: * Human Evaluation Scope: The human evaluation was limited to 104 dialogues and 3 annotators, potentially limiting the robustness of correlation analysis. All evaluators were non-native English speakers, which might have biased naturalness assessments. * Persona Diversity: The human evaluation used only two base personas, restricting the ability to fully assess the approach across a broader range of customer types. * Metric Challenges: Bot naturalness and proactiveness proved challenging to assess consistently across both human and LM evaluators, suggesting a need for clearer definitions or alternative measurement approaches. * Simulator Data: The synthetic data generator occasionally produced outdated product information or lacked clear distinctions between device costs and monthly payments. * Comparative Analysis: The study lacked a direct comparison between persona-driven and non-persona dialogues.

Future Work: * Human Participant Validation: Validate simulation findings with human participants to assess user success rates and satisfaction. * Simulator Improvements: Incorporate up-to-date product information, develop structured budget representations, and ensure better alignment between goals and scenarios. * Broader Use Cases: Extend the framework to other domains beyond smartphone sales. * More Comprehensive Evaluation: Conduct larger-scale human evaluations and comparative analyses.

Summary updated

BibTeX
@inproceedings{gromada2025evaluating,
  author = {Justyna Gromada and Alicja Kasicka and Ewa Komkowska and Lukasz Krajewski and Natalia Krawczyk and Morgan Veyret and Bartosz Przybył and Lina M. Rojas-Barahona and Michał K. Szczerbak},
  title = {{Evaluating Conversational Agents with Persona-driven User Simulations based on Large Language Models: A Sales Bot Case Study}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure