Why Can't You Convince Me? Modeling Weaknesses in Unpersuasive Arguments

Isaac Persing, Vincent NgIJCAI 2017 · DOI

TL;DR

This paper tackles the challenge of not just predicting how persuasive an argument is, but why it fails to persuade. They introduce a new dataset of annotated debate comments and a novel approach that models persuasiveness by identifying specific error types, outperforming existing methods and providing actionable feedback for argument improvement.

What Problem Does This Paper Address?

In the field of computational argumentation, a significant focus has been on determining how persuasive an argument is. However, simply knowing an argument is unpersuasive isn't enough for an author to improve it. To make an argument more effective, one needs to understand the specific weaknesses or errors that undermine its persuasiveness. This paper addresses this gap by aiming to not only score an argument's persuasiveness but also to identify the underlying reasons for its lack of impact. This is crucial for applications like automated writing feedback systems, where providing constructive criticism is key.

Methods

The authors propose a multi-faceted approach:

  • Annotation and Error Taxonomy: They created a new dataset by annotating 1,208 debate comments. Crucially, these annotations include not only a persuasiveness score but also the presence of five specific error types that can negatively impact an argument: Grammar Errors (GE), Lack of Objectivity (LO), Inadequate Support (IS), Unclear Assertion (UA), and Unclear Justification (UJ).
  • Dual Regression Task: The core of their method treats the problem as six independent regression tasks: one for predicting the overall persuasiveness score (AP) and five for predicting the severity of each error type.
  • Feature Engineering: They developed 11 distinct feature types to represent each argument. These features capture various aspects, including grammar error frequency, subjectivity indicators, definiteness, pronoun usage, citation counts, content lemma overlap, assertion and justification length, subject-argument matching, and strong thesis statement indicators.
  • Machine Learning Model: They employ Support Vector Regression (SVR) models, trained using the LIBSVM library, to perform the prediction tasks.

Datasets

  • IDEA Debate Corpus: The authors utilized a subset of 165 debates from the International Debate Education Association (IDEA) website. This corpus consists of 1,208 arguments, each with a motion, an assertion, and a justification.
  • New Annotated Dataset: The paper introduces a newly annotated dataset derived from the IDEA corpus. This dataset includes:
  • Argument Persuasiveness (AP) scores (on a 1-6 scale).
  • Scores for five error classes (GE, LO, IS, UA, UJ).
  • This dataset is made publicly available to the NLP community.

Key Insights and Ideas

  • Errors as Explanations: The central idea is that unpersuasiveness can be largely explained by specific, identifiable errors. This moves beyond simply scoring persuasiveness to providing diagnostic feedback.
  • Feature Importance: The study reveals that the five identified error classes have significant negative impacts on argument persuasiveness, suggesting that addressing these specific issues is key to improving arguments.
  • Error Prediction as a Proxy: The research demonstrates that predicting persuasiveness using the predicted error scores (EF features) yields results comparable to, and sometimes better than, using the original features directly (SF features). This validates the explanatory power of the error models.
  • Context Matters (but not always): While some prior work focused on conversational context, this paper shows that arguments written in response to a specific debate topic (rather than a preceding comment) can be understood and evaluated independently, simplifying the task.

Key Results

  • Outperforming Baselines: The proposed approach significantly outperforms six baseline systems on the argument persuasiveness scoring task across all three evaluation metrics (E, ME, PC).
  • Effective Error Prediction: The system also achieves strong performance on predicting the five error severity tasks, often outperforming baselines.
  • Error-Based Persuasiveness: When using predicted error scores as features for persuasiveness prediction (EF features), the performance is comparable to or better than using the original features (SF features), indicating that the errors capture most of the relevant information.
  • Learning Curves: Even with limited training data (as little as 20%), the system's performance using error features remains competitive and outperforms baselines.

Limitations and Future Work

  • User Study Needed: The authors acknowledge that while their system identifies errors, a user study is needed to determine how useful these identified errors are for helping authors actually improve their arguments.
  • Inter-Annotator Agreement: While not explicitly detailed as a limitation, the paper presents inter-annotator disagreement levels, suggesting that annotation can be subjective, which is a common challenge in such tasks.
  • Future Work: The primary future direction is the user study mentioned above. Additionally, the authors suggest that improving the predictiveness of their error classifiers could further enhance the overall EF performance.

Summary updated

BibTeX
@inproceedings{persing2017why,
  author = {Isaac Persing and Vincent Ng},
  title = {{Why Can't You Convince Me? Modeling Weaknesses in Unpersuasive Arguments}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/570},
}
Figure