Abstract: Automatic detection of persuasion in online discussion is key to understanding how social media is used. Predicting persuasiveness is difficult, however, due to the need to model world knowledge, dialogue, and sequential reasoning. We focus on modeling the sequence of arguments in social media posts using neural models with embeddings for words, discourse relations, and semantic frames. We demonstrate significant improvement over prior work in detecting successful arguments. We also present an error analysis assessing novice human performance at predicting persuasiveness.
Introduction
Predicting whether an argument will be persuasive is a "grand challenge" in Natural Language Processing (NLP). It requires more than just a surface-level understanding of vocabulary; it involves modeling world knowledge, social dynamics, and complex reasoning. While prior research has focused on "interplay"—the vocabulary overlap between a rebuttal and the original post—this paper addresses a critical structural gap. The authors argue that how an argument is sequenced (e.g., starting with evidence, making a polite concession, and ending with a rhetorical question) is just as vital as the specific words used.
The CMV Dataset
To study this phenomenon, the researchers utilized data from the "Change My View" (CMV) subreddit. CMV is a unique forum where users post an opinion and invite others to challenge it. If a user's view is successfully changed, they award a "delta" (Δ) symbol to the persuasive post. This provides a naturally labeled, gold-standard dataset of successful versus unsuccessful persuasion.
Modeling Argument Sequences
The authors developed a hierarchical deep learning architecture designed to capture the flow of logic and rhetoric. The model represents documents through several layers:
- Sentence Representation: Each sentence is encoded using GloVe word embeddings for semantic meaning, FrameNet semantic frames for event types, and PDTB discourse relations to model how sentences relate to one another (e.g., comparison or concession).
- Highway Networks: These allow the model to adaptively integrate disparate word, frame, and discourse data.
- Hierarchical LSTM with Attention: A Long Short-Term Memory (LSTM) network processes the sequence of sentences, while an attention mechanism identifies which sentences are most influential for the final prediction.
- Dynamic Memory Network (DMN): This component iteratively compares the response to the original post to find abstract representations of their interaction.
Results
The findings confirm that sequence matters significantly. The "all-LSTM" models consistently outperformed non-sequential baselines. Specifically, removing the LSTM component led to a sharp drop in performance, proving that the order of arguments provides a signal that words alone cannot capture.
As shown in the results, the best-performing model (all-LSTM+memory+IP) achieved an accuracy of 81.0% on the "full path" task, a substantial improvement over the Bag-of-Words (BoW) baseline of 61.9%. In a pairwise task—predicting which of two responses to the same post was more persuasive—the model reached 71.9% accuracy.
The Human Performance Gap
One of the most surprising insights from the study was the comparison between the neural model and novice human performance. Human annotators achieved roughly 57% accuracy on influence prediction, while the best model surpassed 80%. This suggests that persuasion is subtle and that humans may be biased by their own opinions when judging the persuasiveness of an argument.
Furthermore, the model (and humans) struggled most with topics like "Government" and "Economics." These areas often require deep external world knowledge or empirical facts that are not explicitly contained within the text of the post, highlighting a current limitation of text-only models.
Conclusion and Future Work
This research demonstrates that modeling the structural flow of an argument is a powerful tool for detecting persuasion. While the current model excels at identifying if a view changed, future work aims to explore why it changed. The authors suggest incorporating world knowledge components to improve performance on factual topics and exploring unsupervised learning to identify persuasive patterns in data without explicit labels. Moving forward, modeling the back-and-forth of threads as multi-party dialogues could further refine our understanding of online influence.
Summary updated