Weakly-Supervised Hierarchical Models for Predicting Persuasive Strategies in Good-faith Textual Requests

Jiaao Chen, Diyi Yang

Abstract

Modeling persuasive language has the potential to better facilitate our decision-making processes. Despite its importance, computational modeling of persuasion is still in its infancy, largely due to the lack of benchmark datasets that can provide quantitative labels of persuasive strategies to expedite this line of research. To this end, we introduce a large-scale multi-domain text corpus for modeling persuasive strategies in good-faith text requests. Moreover, we design a hierarchical weakly-supervised latent variable model that can leverage partially labeled data to predict such associated persuasive strategies for each sentence, where the supervision comes from both the overall document-level labels and very limited sentence-level labels. Experimental results showed that our proposed method outperformed existing semi-supervised baselines significantly.

Introduction

Persuasion is a cornerstone of human interaction, yet its computational modeling remains a significant challenge. The primary bottleneck is the scarcity of fine-grained data; labeling every sentence in a request with specific strategies—such as "appealing to emotion" or "providing evidence"—is labor-intensive and expensive. While document-level labels (e.g., whether a request was successful) are often readily available in online platforms, mapping these outcomes back to specific sentence-level strategies requires sophisticated modeling. Furthermore, existing research often focuses on single domains, limiting the generalizability of persuasion theories across different contexts.

Methods: The WS-VAE Architecture

To address the lack of sentence-level labels, we propose a Weakly-Supervised Hierarchical Latent Variable Model (WS-VAE). This architecture bridges the gap between fine-grained sentence strategies and coarse document-level outcomes through a multi-level approach:

  1. Sentence-Level VAE: For each sentence, the model learns two distinct latent variables: $y$, representing the discrete persuasion strategy, and $z$, representing the remaining context or content. By disentangling these, the model isolates the persuasive signal from the general narrative.
  2. Document-Level Predictor: The model aggregates these latent variables across all sentences in a document to predict the overall outcome (e.g., success or failure of the request).
  3. Weak Supervision: The model leverages a small set of fully labeled sentences alongside a much larger pool of unlabeled text. It uses the document-level success label as an indirect signal to refine its understanding of which sentence-level strategies contribute to successful outcomes.
The hierarchical WS-VAE architecture, showing the disentanglement of strategy (y) and content (z) at the sentence level to predict document labels.

Multi-Domain Dataset

We introduce a large-scale, multi-domain corpus consisting of over 100,000 "good-faith" requests from three distinct platforms:

  • Kiva: A peer-to-peer philanthropic lending platform.
  • r/Random_Acts_Of_Pizza (RAOP): A subreddit for requesting free pizza for social or personal reasons.
  • r/borrow: A subreddit for small, interest-bearing peer-to-peer loans.

We unified various persuasion theories into eight core strategies: Commitment, Emotion, Politeness, Reciprocity, Scarcity, Credibility, Evidence, and Impact. Approximately 5% of the data was manually annotated to create a gold-standard evaluation set.

Key Insights and Results

The WS-VAE-BERT model, which utilizes a pre-trained BERT encoder, consistently outperformed standard supervised and semi-supervised baselines. On the Kiva dataset, with only 20 labeled documents, it achieved a Macro F1 of 34.2, significantly higher than standard LSTM (26.1) or vanilla BERT (28.6) approaches.

Strategy Importance vs. Frequency

Our analysis revealed a compelling distinction between strategy frequency and strategy impact. While "Evidence" is a frequent strategy across datasets, it often carries lower weight in predicting success compared to more heuristic cues.

Attention weights for different strategies across the three datasets, indicating their relative importance in predicting request success.

As shown in the attention weight analysis, "superficial" strategies like Reciprocity (promising to pay it forward) and Scarcity (emphasizing urgency) often had higher weights for success than "deeper" logical strategies like Evidence. This suggests that in fast-paced online environments, quick heuristic cues can be more influential than long-form logical arguments. Additionally, while content ($z$) generally carries more weight in predicting success than the specific strategy ($y$), the right strategy provides a necessary boost to the request's effectiveness.

Conclusion

This work demonstrates that indirect supervision from document-level outcomes can effectively guide the discovery of sentence-level persuasive strategies. By disentangling strategy from content, our model provides a more nuanced understanding of how people persuade others in online communities. Future work will focus on integrating audience factors, such as requester history and social standing, to provide a more holistic view of persuasive success.

Summary updated

Abstract

Modeling persuasive language has the potential to better facilitate our decision-making processes. Despite its importance, computational modeling of persuasion is still in its infancy, largely due to the lack of benchmark datasets that can provide quantitative labels of persuasive strategies to expedite this line of research. To this end, we introduce a large-scale multi-domain text corpus for modeling persuasive strategies in good-faith text requests. Moreover, we design a hierarchical weakly-supervised latent variable model that can leverage partially labeled data to predict such associated persuasive strategies for each sentence, where the supervision comes from both the overall document-level labels and very limited sentence-level labels. Experimental results showed that our proposed method outperformed existing semi-supervised baselines significantly. We have publicly released our code at https://github.com/GT-SALT/Persuasion_Strategy_WVAE.
BibTeX
@inproceedings{chen2021weakly,
  author = {Jiaao Chen and Diyi Yang},
  title = {{Weakly-Supervised Hierarchical Models for Predicting Persuasive Strategies in Good-faith Textual Requests}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=WE6_271tg4x},
}
Figure