MADS: Multi-Agent Dialogue Simulation for Diverse Persuasion Data Generation

Mingjin Li, Yu Liu, Huayi Liu, Xiang Ye, Chao Jiang, Hongguang Zhang, Yu RuanEMNLP 2025

TL;DR

This paper introduces MADS, a novel multi-agent dialogue simulation framework that uses agent self-play to generate diverse and persuasive conversational data. MADS addresses the critical industry challenge of acquiring high-quality, personalized training data for dialogue systems, especially in cold-start scenarios, by enabling scalable, low-cost data generation without extensive human annotation.

What Problem Does This Paper Address?

Developing effective persuasive dialogue systems is crucial for applications in marketing, healthcare, and finance. However, building these systems faces significant hurdles: a lack of authentic, multi-turn dialogue data with rich user profiles, difficulties in evaluating systems in cold-start scenarios where interaction logs are scarce, and the inefficiency and brittleness of manually crafted prompts. Existing approaches often rely heavily on real user data or human feedback, limiting their scalability and practicality. Furthermore, current LLM-based dialogue systems struggle with personalization and adapting to diverse user behaviors, especially in strategic business contexts.

Methods

MADS employs a three-agent coordinated system:

  • User Agents: These agents simulate diverse personas by leveraging personality signifiers like Zodiac Signs and MBTI types, along with business attributes. This allows for the generation of varied user behaviors and attitudes.
  • Dialog Agent: This agent engages in multi-turn interactions with User Agents, executing task-oriented persuasion strategies.
  • Optimization Agent: This module automates dialogue annotation and refines the Dialog Agent's prompts. It includes sub-modules for summarizing dialogues, evaluating outcomes, and evolving the agent's strategies.

The core of MADS is a self-optimizing, closed-loop process where simulated dialogues feed into a platform-based LLM training pipeline. This iterative process refines prompts and generates increasingly effective persuasive dialogues. A key concept is Chain-of-Attitude (CoA) modeling, which tracks user attitude changes throughout the dialogue, providing a structured way to measure persuasive success.

Datasets

The paper utilizes the DailyPersuasion dataset for domain-targeted sampling to compare MADS with existing benchmarks. They also generate synthetic dialogue data for an insurance scenario to fine-tune and evaluate their models.

Key Insights and Ideas

  • Personality Signifiers Drive Diversity: Incorporating symbolic personality traits like Zodiac Signs and MBTI types into User Agent prompts significantly enhances the diversity of simulated user behaviors and attitude trajectories, even with constant Dialog Agent prompts.
  • Business Attributes Enhance Realism: Business attributes, when combined with personality traits, lead to the most distinct Chain-of-Attitude distributions, suggesting a more nuanced and realistic simulation of user responses in business contexts.
  • MADS Data Boosts Small LLMs: Fine-tuning smaller LLMs with MADS-generated data significantly improves their persuasive capabilities, demonstrating the value of this synthetic data for enhancing model performance.
  • Self-Optimization is Key: The iterative self-optimization mechanism within MADS is crucial for refining prompts and improving task completion rates, especially in scenarios where initial prompts are not fully effective.

Key Results

  • MADS significantly improved the persuasion capacity of small LLMs, leading to a 22.4% increase in organic traffic conversion rate (from 1.83% to 2.24%) in a real-world marketing scenario.
  • Incorporating personality traits (Zodiac Signs, MBTI) led to a 35.37% increase in average attitude entropy compared to a baseline with only demographic attributes.
  • The MADS framework achieved over a 28% improvement in user intent rate when deployed in real-world business settings.
  • In the Make Me Pay (MMP) evaluation, fine-tuning with MADS data increased the donation success rate for ERNIE-Lite from 18% to 30% and reduced user withdrawal rate from 70% to 58%.

Limitations and Future Work

  • Single-Dimension Evaluation: The current evaluation primarily focuses on persuasive effectiveness. Future work should incorporate a broader set of metrics, including factual accuracy, emotional appropriateness, personalization, coherence, and ethical alignment, for a more holistic assessment.
  • Fixed Attitude Taxonomy: The use of a predefined, fixed taxonomy for user attitudes might introduce representational bias. Future research could explore data-driven or adaptive attitude representations to capture a wider range of user variability.
  • Prompt Optimization Granularity: The current prompt evolution assumes a single prompt strategy can accommodate diverse personas. Future work could explore a more layered approach, stratifying prompt evolution within user clusters to better capture intra-group coherence and inter-group diversity.
  • Alignment with Human Judgment: The reliance on automatic metrics for evaluation may not always align with human judgment. Incorporating expert or crowd-sourced human evaluation would provide a more robust validation of practical effectiveness.
  • Dependency on User Profiling Systems: The framework's performance is dependent on its ability to interface with existing user profiling systems. Future work needs to address how MADS can generalize or adapt without reliable profile grounding.

Summary updated

BibTeX
@inproceedings{li2025mads,
  author = {Mingjin Li and Yu Liu and Huayi Liu and Xiang Ye and Chao Jiang and Hongguang Zhang and Yu Ruan},
  title = {{MADS: Multi-Agent Dialogue Simulation for Diverse Persuasion Data Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure