Introduction
Argument mining (AM) has traditionally focused on structured environments like legal documents or formal debates. However, the rise of social media has created a need for tools capable of navigating the "wild west" of informal communication, where arguments are often obscured by slang, hashtags, and emoticons. This research expands the scope of AM on Twitter by moving beyond simple argument detection. The authors propose a framework to distinguish factual arguments from subjective opinions and automatically identify the specific sources cited within factual claims, a critical step for automated fact-checking and provenance verification.
Methodology
The researchers implement a three-step pipeline to process argumentative content:
- Argument Detection: A binary classification task to separate argumentative tweets from non-argumentative ones.
- Factual vs. Opinion Classification: Argumentative tweets are further categorized as either Factual (containing provable information or reported speech) or Opinion (purely subjective expressions).
- Source Identification: For tweets identified as factual, the system attempts to extract the information source (e.g., news agencies or specific individuals).
Technical Approach
The study utilized supervised learning models, specifically Logistic Regression (LR) and Random Forests (RF). Feature engineering focused on four distinct categories: lexical (unigrams, bigrams, WordNet synsets), Twitter-specific (punctuation, emoticons), syntactic/semantic (POS tags, dependency relations), and sentiment (polarity scores via Alchemy API). Source extraction was handled through a hybrid approach combining string matching against a gazetteer of news agencies and Named Entity Recognition (NER) linked to DBpedia.
Data Collection and Annotation
The study utilized two primary datasets focused on European political crises, chosen for their naturally high density of argumentative structures:
- DART (#Grexit): An existing dataset of 987 tweets related to the Greek debt crisis.
- #Brexit Dataset: A new dataset of 900 tweets collected and annotated by the authors following the DART methodology.
Both datasets were manually annotated for the three pipeline tasks, achieving high inter-annotator agreement with Kappa scores ranging between 0.72 and 0.76.
Experimental Results
The best-performing models generally utilized Logistic Regression with the full feature set.
Argument Detection
The system achieved an overall F1-score of 0.78. However, performance varied significantly by class; the model was much more effective at identifying arguments (0.85 F1) than non-arguments (0.52 F1).
Factual vs. Opinion Classification
This task achieved an F1-score of 0.80. Similar to the first task, there was a performance gap between classes: the model was significantly better at identifying opinions (0.88 F1) than factual claims (0.50 F1). This is likely due to the high prevalence of subjective language and the relative scarcity of factual claims in the Twitter domain.
Source Identification
The hybrid matching and heuristics approach achieved an F1-score of 0.67, significantly outperforming a baseline NER-only approach, which only reached an F1 of 0.33.
Discussion and Future Work
A key insight from the study is the importance of "reported speech." An argument is classified as factual if it claims to be a fact or quotes an external entity (e.g., "Hollande urges UK to start Brexit talks"), regardless of the underlying truth of the claim. This distinction is vital for downstream verification systems.
However, several challenges remain. Sarcasm and irony continue to be primary drivers of misclassification, as linguistically complex tweets often trip up standard classifiers. Furthermore, the factual argument dataset remains relatively small (368 tweets), limiting the potential for more complex deep learning models. Future work will focus on link verification—following shortened URLs to ensure cited sources are legitimate—and integrating dedicated sarcasm detection modules to improve the precision of the factual/opinion split.
Summary updated