Introduction
Stance detection—identifying whether a text is pro, con, or neutral toward a specific topic—is a critical task in natural language processing. Traditionally, this has been approached as a supervised learning problem, requiring extensive labeled datasets for every new topic. This paper introduces a fully unsupervised, topic-independent method that leverages the logical structure of "arguments from consequences" (e.g., "If we do X, then Y happens"). By decomposing claims into structured "effect triples," the proposed model achieves performance comparable to fine-tuned BERT models without any labeled training data.
The Challenge of Stance Detection
A significant bottleneck in stance detection is the need for manual annotation whenever a new debate topic emerges. Furthermore, stance is distinct from sentiment. For instance, the sentence "Holocaust denial harms survivors" carries a negative sentiment but represents a positive stance toward the topic "Criminalization of Holocaust denial." Standard sentiment analysis often fails to capture this nuance. This research addresses these issues by focusing on the underlying logic of argumentation rather than topic-specific keywords.
Methodology: The Effect Triple
The core innovation of this work is the Effect Triple, defined as:
$$\langle (T, dir), (P, eff), (O, sent) \rangle$$
- Target ($T, dir$): The topic and whether the claim suggests increasing ($+1$) or decreasing ($-1$) it.
- Predicate ($P, eff$): The action verb and its effect on the object (e.g., "improves" is $+1$, "damages" is $-1$).
- Object ($O, sent$): The result and its inherent sentiment (e.g., "health" is $+1$, "corruption" is $-1$).
The final stance is calculated by multiplying these signs: $Stance = dir \times eff \times sent$.
To implement this pipeline, the researchers utilized:
- Extended Connotation Frames (ECF): A lexicon of approximately 2,500 verbs indicating positive or negative effects.
- Sentiment Lexicons: Standard lexicons (e.g., MPQA) to evaluate the sentiment of objects.
- Dependency Parsing: To extract relationships between the target, verb, and object using specific linguistic patterns.
Experimental Setup
The model was evaluated on two primary datasets:
- Debatepedia: 1,894 arguments across 190 topics, annotated for stance and consequence-based logic.
- Wiki Dataset: A benchmark of 2,394 claims from Wikipedia to test generalizability across different text styles.
Key Results
The unsupervised system, particularly when using the ECF lexicon, demonstrated strong performance:
- Competitive with BERT: On consequence-based arguments in Debatepedia, the system achieved a Macro-F1 of 0.73, nearly matching the 0.74 achieved by a fine-tuned BERT model.
- Superior Robustness: On non-consequence arguments, the unsupervised system (F1 0.63) actually outperformed BERT (F1 0.60).
- Predictability: The model showed significantly lower variance (standard deviation ~0.03-0.07) compared to BERT (~0.13-0.20), suggesting more stable performance on unseen data.
Discussion and Insights
By targeting the structure of an argument (If A leads to B, and B is good, then A is good), the model bypasses the need for topic-specific training, making it "zero-shot" ready. Unlike "black-box" neural networks, this method is inherently interpretable; errors can be traced back to specific components of the triple (e.g., a misidentified sentiment). This approach successfully disentangles stance from sentiment by explicitly modeling the direction of effects.
Limitations and Future Work
While effective, the system currently relies on a fixed set of dependency patterns, which can struggle with complex or idiomatic sentences. Lexicon gaps for ambiguous words and challenges in high-level target identification remain areas for improvement. Future work aims to expand this framework to other argumentation types, such as arguments from expert opinion or example.
Summary updated