Seamlessly Integrating Factual Information and Social Content with Persuasive Dialogue

Maximillian Chen, Weiyan Shi, Feifan Yan, Ryan Hou, Jingwen Zhang, Saurav Sahay, Zhou Yu

Abstract

Persuasive dialogue requires a delicate balance between goal-oriented task completion and the maintenance of social rapport. Existing end-to-end (E2E) dialogue systems often fail in this regard, oscillating between robotic factual delivery and aimless social interaction. This paper introduces the Response-Agenda Pushing (RAP) framework, a modular architecture designed to decouple social acknowledgment and factual accuracy from the core persuasive strategy. By utilizing a specialized dispatcher and a novel loss function ($L_p$) to ensure strategy adherence, RAP achieves significantly higher user engagement and perceived competence. Experimental results from a double-blind study show that RAP increases user response length by over 50% and outperforms baseline models across all perceived human-like traits.

Introduction

Persuasion is a complex social interaction that necessitates building trust, demonstrating empathy, and acknowledging a counterpart's perspective while steering the conversation toward a specific goal. In the context of digital assistants, this presents a significant challenge. Most current systems fall into two categories: strictly task-oriented bots that lack social nuance, or purely social bots that struggle to maintain a strategic agenda.

In persuasive scenarios—such as soliciting charitable donations—a lack of social acknowledgment can destroy a bot's credibility. If a user shares a personal sentiment regarding poverty and the bot immediately pivots to a donation request without acknowledgment, the interaction feels transactional and insensitive. This research addresses the challenge of interweaving social content and factual accuracy without losing sight of the conversational objective.

The RAP Framework

The authors propose the Response-Agenda Pushing (RAP) framework, which replaces monolithic end-to-end models with a modular architecture. This separation of concerns allows the system to handle diverse conversational requirements simultaneously.

Modular Architecture

  1. The Dispatcher: Acts as the central controller, classifying user utterances to determine if the system should provide a factual answer, a social response, or proceed with the persuasive agenda.
  2. Factual Answer Module: Utilizes Sentence-BERT embeddings to retrieve relevant information from training data, ensuring factual consistency and preventing the hallucinations common in generative models.
  3. Social Response Module: Employs BlenderBot 2.0 (3B parameters) to generate empathetic, zero-shot social responses that validate the user's input.
  4. Persuasive Agenda-Pushing Module: Built on a fine-tuned BART model, this module manages the strategic direction of the conversation.

Strategic Adherence

A key technical innovation in RAP is the introduction of a specialized loss function ($L_p$) during the fine-tuning of the BART-based agenda pusher. This function penalizes the model if it generates an utterance that deviates from the intended "dialogue act" (e.g., providing a greeting when the strategy called for a personal story). This ensures the system remains strategically consistent throughout the interaction.

Methodology and Data

The framework was developed and evaluated using the PERSUASIONFORGOOD dataset, which consists of 1,017 human-human conversations centered on persuading individuals to donate to Save the Children. A subset of 300 conversations includes sentence-level annotations for dialogue acts such as "Credibility Appeal," "Emotional Appeal," and "Self-modeling."

To evaluate RAP, the researchers conducted a double-blind study with 108 participants, comparing the modular framework against a baseline BART model. Participants interacted with the bots and subsequently rated them on various Likert-scale metrics.

Key Results

Enhanced User Engagement

RAP demonstrated a superior ability to keep users engaged. Users responding to the RAP framework wrote significantly more than those interacting with the baseline. On average, users contributed 5.75 words per utterance for RAP, compared to 3.70 words for the baseline model.

Perceived Competence and Persuasiveness

RAP outperformed the baseline across every qualitative metric measured (on a 1-5 scale):

  • Competence: 3.00 (RAP) vs. 2.53 (Baseline)
  • Confidence: 3.72 (RAP) vs. 3.35 (Baseline)
  • Persuasiveness: 2.72 (RAP) vs. 2.63 (Baseline)

Furthermore, the specialized loss function significantly improved strategy adherence. The BART model with the $L_p$ penalty reached 62.38% accuracy in following the intended persuasive strategy, a substantial improvement over the <30% accuracy observed in models without the penalty.

Discussion and Insights

The Role of Social Acknowledgment

A primary insight from this study is that social "chit-chat" is not a distraction from the task but a prerequisite for success. By acknowledging user statements (e.g., "I agree with you" or "That is a great point") before pivoting to factual information, the bot maintained higher levels of user rapport.

The Friendliness-Competence Loop

Interestingly, the data suggests that factual accuracy contributes to perceived friendliness. When the bot answered factual questions correctly, users perceived it as more pro-social and helpful. This indicates that in persuasive contexts, competence and social warmth are deeply intertwined.

Limitations and Future Work

Despite its successes, the RAP framework faces several challenges:

  • Temporal Knowledge Gaps: Because the underlying models (BlenderBot and BART) have fixed training cut-offs, the system occasionally struggles with current events (e.g., recent geopolitical conflicts), leading to tone-deaf responses when users bring up modern contexts.
  • Classifier Bottlenecks: The system's overall performance is currently limited by the dialogue act classifier's F1-score (0.66). Improving this classification accuracy is a priority for future development.
  • Dynamic Strategy Planning: Currently, the bot follows a fixed sequence of high-probability strategies. Future iterations could incorporate a dynamic planner that adjusts persuasive appeals based on real-time assessments of user personality or resistance levels.

Summary updated

BibTeX
@inproceedings{chen2022seamlessly,
  author = {Maximillian Chen and Weiyan Shi and Feifan Yan and Ryan Hou and Jingwen Zhang and Saurav Sahay and Zhou Yu},
  title = {{Seamlessly Integrating Factual Information and Social Content with Persuasive Dialogue}},
  booktitle = {AACL/IJCNLP},
  year = {2022},
  url = {https://openreview.net/forum?id=pcNebunmfI2},
}
Figure