Abstract
Stance detection in social media is a complex task that requires identifying an author's position toward a specific target. This study compares a traditional feature-engineered logistic regression model against a modern Convolutional Neural Network (CNN) using the SemEval-2016 Task 6 dataset. While the CNN demonstrated superior performance during internal cross-validation, the feature-based model proved more robust on official test data. Our findings suggest that for small-scale datasets, leveraging external knowledge through lexicons and crawled data can outperform deep learning architectures that are prone to overfitting.
Introduction
Stance detection aims to determine whether a text expresses a FAVOR, AGAINST, or NONE (neutral or irrelevant) stance toward a given target, such as "Climate Change" or "Hillary Clinton." This task is distinct from sentiment analysis; for instance, a user might express a negative sentiment toward current laws to indicate a positive stance toward legal reform. Tweets exacerbate these difficulties due to their brevity, informal syntax, and frequent use of sarcasm. This research investigates whether automated feature extraction via CNNs can surpass hand-crafted features and external knowledge bases in this challenging domain.
Methodology
The researchers developed two primary pipelines to evaluate performance:
1. Feature-Based Approach
This model utilized a Logistic Regression classifier built upon three categories of features:
- Linguistic Features: Standard bag-of-words, part-of-speech (POS) tags, and dependency relations.
- Sentiment Lexicons: Integration of SentiWordNet to provide quantitative sentiment scores for individual words.
- HighPMI Features: To compensate for the small training set, the team crawled millions of external tweets related to the target topics. They used Point-wise Mutual Information (PMI) to identify words strongly associated with specific stances (e.g., "humanist" for "atheism").
2. Convolutional Neural Network (CNN) Approach
The CNN architecture followed the design proposed by Yoon Kim (2014). It utilized word embeddings pre-trained on Wikipedia using the Continuous Bag-of-Words (CBOW) model.
The model consists of a single convolution layer with 200 filters, followed by a max-pooling layer and a fully connected layer with a softmax output for classification.
The "Two-Step" Strategy
For both models, a hierarchical approach proved most effective. Instead of a direct three-way classification, the system first determined if a tweet was relevant to the target (Topic Classifier) and then determined its polarity (Two-way Polarity Classifier).
Results and Analysis
The performance was measured using the macro-average of F1-scores for the FAVOR and AGAINST classes. During 10-fold cross-validation on training data, the CNN-Best model achieved an F1 of 0.6831, comfortably beating the Feature-Best model's 0.6297.
However, on the official test data, the results flipped: the Feature-Based model achieved 0.6426, while the CNN dropped to 0.6221. This "overfitting trap" suggests the CNN memorized patterns specific to the small training set (395–789 tweets per topic) that were not present in the broader test set.
Visualizing Model Focus
To understand the CNN's decision-making process, the authors generated heatmaps of word importance.
The visualization revealed that while the CNN successfully identified topic-relevant terms (e.g., "movement") and sentiment-laden words (e.g., "hurt," "help"), it struggled to correctly interpret negation (e.g., "not") within the context of stance.
Conclusion
The study concludes that while CNNs offer powerful automated feature extraction, they are highly sensitive to training data size. The feature-based model's success on the official test data highlights the value of "world knowledge" provided by external lexicons and large-scale crawled data. Future work should focus on hybrid models that combine the representational power of neural networks with the robustness of external linguistic resources.
Summary updated