Tohoku at SemEval-2016 Task 6: Feature-based Model versus Convolutional Neural Network for Stance Detection

Yuki Igarashi, Hiroya Komatsu, Sosuke Kobayashi, Naoaki Okazaki, Kentaro Inui

Abstract

Stance detection in social media is a complex task that requires identifying an author's position toward a specific target. This study compares a traditional feature-engineered logistic regression model against a modern Convolutional Neural Network (CNN) using the SemEval-2016 Task 6 dataset. While the CNN demonstrated superior performance during internal cross-validation, the feature-based model proved more robust on official test data. Our findings suggest that for small-scale datasets, leveraging external knowledge through lexicons and crawled data can outperform deep learning architectures that are prone to overfitting.

Introduction

Stance detection aims to determine whether a text expresses a FAVOR, AGAINST, or NONE (neutral or irrelevant) stance toward a given target, such as "Climate Change" or "Hillary Clinton." This task is distinct from sentiment analysis; for instance, a user might express a negative sentiment toward current laws to indicate a positive stance toward legal reform. Tweets exacerbate these difficulties due to their brevity, informal syntax, and frequent use of sarcasm. This research investigates whether automated feature extraction via CNNs can surpass hand-crafted features and external knowledge bases in this challenging domain.

Methodology

The researchers developed two primary pipelines to evaluate performance:

1. Feature-Based Approach

This model utilized a Logistic Regression classifier built upon three categories of features:

  • Linguistic Features: Standard bag-of-words, part-of-speech (POS) tags, and dependency relations.
  • Sentiment Lexicons: Integration of SentiWordNet to provide quantitative sentiment scores for individual words.
  • HighPMI Features: To compensate for the small training set, the team crawled millions of external tweets related to the target topics. They used Point-wise Mutual Information (PMI) to identify words strongly associated with specific stances (e.g., "humanist" for "atheism").

2. Convolutional Neural Network (CNN) Approach

The CNN architecture followed the design proposed by Yoon Kim (2014). It utilized word embeddings pre-trained on Wikipedia using the Continuous Bag-of-Words (CBOW) model.

Architecture of the CNN model used for stance detection, featuring a single convolution layer and max-pooling.

The model consists of a single convolution layer with 200 filters, followed by a max-pooling layer and a fully connected layer with a softmax output for classification.

The "Two-Step" Strategy

For both models, a hierarchical approach proved most effective. Instead of a direct three-way classification, the system first determined if a tweet was relevant to the target (Topic Classifier) and then determined its polarity (Two-way Polarity Classifier).

Results and Analysis

The performance was measured using the macro-average of F1-scores for the FAVOR and AGAINST classes. During 10-fold cross-validation on training data, the CNN-Best model achieved an F1 of 0.6831, comfortably beating the Feature-Best model's 0.6297.

However, on the official test data, the results flipped: the Feature-Based model achieved 0.6426, while the CNN dropped to 0.6221. This "overfitting trap" suggests the CNN memorized patterns specific to the small training set (395–789 tweets per topic) that were not present in the broader test set.

Visualizing Model Focus

To understand the CNN's decision-making process, the authors generated heatmaps of word importance.

Heatmap visualization showing the words prioritized by the Topic and Two-way Polarity classifiers.

The visualization revealed that while the CNN successfully identified topic-relevant terms (e.g., "movement") and sentiment-laden words (e.g., "hurt," "help"), it struggled to correctly interpret negation (e.g., "not") within the context of stance.

Conclusion

The study concludes that while CNNs offer powerful automated feature extraction, they are highly sensitive to training data size. The feature-based model's success on the official test data highlights the value of "world knowledge" provided by external lexicons and large-scale crawled data. Future work should focus on hybrid models that combine the representational power of neural networks with the robustness of external linguistic resources.

Summary updated

BibTeX
@inproceedings{igarashi2016tohoku,
  author = {Yuki Igarashi and Hiroya Komatsu and Sosuke Kobayashi and Naoaki Okazaki and Kentaro Inui},
  title = {{Tohoku at SemEval-2016 Task 6: Feature-based Model versus Convolutional Neural Network for Stance Detection}},
  booktitle = {SemEval @ NAACL},
  year = {2016},
  doi = {10.18653/v1/s16-1065},
}
Figure