Identifying Appropriate Support for Propositions in Online User Comments

Joonsuk Park, Claire Cardie

Introduction

In the unstructured landscape of internet comment sections, arguments often lack the necessary evidence to be persuasive or informative. While natural language processing (NLP) has made significant strides in extracting arguments from structured texts like legal documents or news articles, online user comments present a messier, more subjective challenge. Researchers from Cornell University addressed this gap in their paper, "Identifying Appropriate Support for Propositions in Online User Comments," by shifting the focus from mere argument extraction to assessing the adequacy of support.

The Problem: Assessing Argument Adequacy

Most existing "argumentation mining" research focuses on identifying the structural components of an argument—conclusions and premises. However, in online forums, such as eRulemaking platforms where citizens comment on proposed federal regulations, arguments are frequently incomplete or poorly substantiated.

The core issue is not just finding an argument, but determining what kind of support a specific claim requires to be considered high-quality. For instance, a value judgment like "The government should ban peanuts on planes" requires a logical reason, whereas a factual claim like "Peanut allergies affect 20% of the population" requires objective evidence. Current tools struggle to distinguish between these types, making it difficult for policy-makers to filter for evidence-based discourse.

Methodology: A Classification Framework

The researchers developed a supervised learning approach using Support Vector Machines (SVM) to classify "propositions"—the smallest units of an argument—into distinct categories based on their verifiability and experiential nature.

Figure 1: Flowchart for classifying propositions into four distinct categories: Non-Argumentative (NonArg), Unverifiable, Verifiable Non-Experiential, and Verifiable Experiential.

As shown in the classification flowchart above, the methodology involved three primary layers:

  1. Annotation Scheme: Sentences and clauses were labeled based on whether they were part of an argument, whether they could be verified with objective evidence, and whether they described personal experiences.
  2. Feature Engineering: The model utilized standard n-grams alongside specialized features:
  3. Verifiability Features: Sentiment clues (subjective words) and "speech event" counts (e.g., "he said").
  4. Experientiality Features: Tense counts (past tense often signals personal narrative) and person counts (1st vs. 3rd person).
  5. Core Clause Tagging (CCT): This technique attempted to use syntactic parsing to isolate the "meat" of a claim from its "wrapper" (e.g., stripping "I believe that..." to focus on the factual proposition).

Dataset and Distribution

The study utilized a gold-standard dataset from Regulation Room, an eRulemaking platform. The dataset comprised 9,476 sentences and clauses from 1,047 comments across two domains: Airline Passenger Rights (APR) and Home Mortgage Consumer Protection (HMCP). Notably, approximately 70% of the propositions were labeled as Unverifiable, highlighting the opinion-heavy nature of public discourse.

Key Results and Performance

The researchers evaluated several feature sets to determine the most effective model for classification. The best-performing model combined unigrams, bigrams, verifiability features, and experientiality features.

Figure 6: Performance metrics (Precision, Recall, and F1) for different feature sets across the three proposition classes. The combined model achieved the highest macro-averaged F1 score of 68.99%.

As detailed in the results table, the model achieved a macro-averaged F1 score of 68.99%. Key findings from the performance data include:

  • Class-Specific Success: The model was highly effective at identifying Unverifiable claims (84.91% F1), likely due to the distinct linguistic markers of subjective opinion.
  • The Challenge of Facts: Identifying Verifiable Non-Experiential claims proved more difficult (52.37% F1). These claims are often characterized by the absence of subjective cues and the use of technical, domain-specific language (e.g., "allergen," "airborne").
  • Feature Impact: The addition of specialized verifiability (VER) and experientiality (EXP) features provided a statistically significant improvement over the simple unigram baseline.

Conclusion and Future Directions

This research establishes a framework for identifying the "receipts" in online arguments. By distinguishing between claims that need logical reasons and those that need hard evidence, AI can begin to evaluate the quality of public participation in policy-making.

Future work aims to move from classifying individual propositions to relational mining—automatically determining if a provided "reason" actually supports a specific "claim." Ultimately, this technology could provide real-time feedback to users, nudging them to provide citations or evidence to strengthen their arguments before they hit "submit."

Summary updated

Abstract

The ability to analyze the adequacy of supporting information is necessary for determining the strength of an argument.1 This is especially the case for online user comments, which often consist of arguments lacking proper substantiation and reasoning. Thus, we develop a framework for automatically classifying each proposition as UNVERIFIABLE, VERIFIABLE NONEXPERIENTIAL, or VERIFIABLE EXPERIENTIAL2, where the appropriate type of support is reason, evidence, and optional evidence, respectively3. Once the existing support for propositions are identified, this classification can provide an estimate of how adequately the arguments have been supported. We build a goldstandard dataset of 9,476 sentences and clauses from 1,047 comments submitted to an eRulemaking platform and find that Support Vector Machine (SVM) classifiers trained with n-grams and additional features capturing the verifiability and experientiality exhibit statistically significant improvement over the unigram baseline, achieving a macro-averaged F1 of 68.99%.
BibTeX
@inproceedings{park2014identifying,
  author = {Joonsuk Park and Claire Cardie},
  title = {{Identifying Appropriate Support for Propositions in Online User Comments}},
  booktitle = {ArgMining @ ACL},
  year = {2014},
  doi = {10.3115/v1/w14-2105},
}
Figure