DeepStance at SemEval-2016 Task 6: Detecting Stance in Tweets Using Character and Word-Level CNNs

Prashanth Vijayaraghavan, Ivan Sysoev, Soroush Vosoughi, Deb Roy

Abstract

This paper describes our approach for the Detecting Stance in Tweets task (SemEval-2016 Task 6). We utilized recent advances in short text categorization using deep learning to create word-level and character-level models. The choice between word-level and character-level models in each particular case was informed through validation performance. Our final system is a combination of classifiers using word-level or character-level models. We also employed novel data augmentation techniques to expand and diversify our training dataset, thus making our system more robust. Our system achieved a macro-average precision, recall and F1-scores of 0.67, 0.61 and 0.635 respectively.

Introduction

Stance detection is the task of determining an author's position—typically "Favor," "Against," or "None"—toward a specific target. While often confused with sentiment analysis, stance detection is uniquely target-specific. For instance, a positive sentiment toward one political figure often implies a negative stance toward their opponent. Detecting these nuances in tweets is challenging due to the short, noisy nature of the text, which frequently includes slang, typos, and idiosyncratic hashtags.

Methodology

Our system, DeepStance, employs two primary deep learning architectures to capture different linguistic granularities:

  1. Character-Level CNN (CharCNN): This model processes text character-by-character, making it inherently robust to misspellings and the informal language prevalent on Twitter.
  2. Word-Embedding CNN: This model utilizes word embeddings to map tokens into a high-dimensional vector space, capturing semantic relationships more effectively on smaller datasets.

Data Augmentation: The "Secret Sauce"

Given the limited size of the official SemEval-2016 dataset, we implemented two novel data augmentation strategies:

  • Hashtag Mining: We used Mutual Information to identify hashtags strongly correlated with specific stances. These hashtags served as proxies to collect 1.7 million additional tweets from historical archives.
  • Word2Vec Substitution: We generated 500,000 synthetic tweets per target by replacing words with contextually similar synonyms identified via Word2Vec embeddings. This approach provided better coverage for informal language than traditional dictionary-based methods.
Table 1: Hashtags identified as strong indicators for specific stances across different targets.

Results and Discussion

DeepStance achieved a macro-average F1-score of 0.635, ranking 8th out of 19 teams overall. Notably, the system secured the #1 rank for the "Legalization of Abortion" target.

A key finding was that the optimal model architecture is target-dependent. For example, character-level models showed significant improvement for the "Hillary Clinton" target when augmented with sufficient data, whereas word-level models were often more efficient for targets with less available training data.

Table 2: Comparison of performance metrics between Word-level and Character-level CNN models across various targets.

Conclusion

The DeepStance system demonstrates that combining character and word-level features with aggressive data augmentation can effectively address the challenges of stance detection in noisy social media text. Future work will explore unsupervised pre-training and more sophisticated "bootstrapping" techniques to further refine the quality of augmented training samples.

Summary updated

Abstract

This paper describes our approach for the Detecting Stance in Tweets task (SemEval-2016 Task 6). We utilized recent advances in short text categorization using deep learning to create word-level and character-level models. The choice between word-level and character-level models in each particular case was informed through validation performance. Our final system is a combination of classifiers using word-level or character-level models. We also employed novel data augmentation techniques to expand and diversify our training dataset, thus making our system more robust. Our system achieved a macro-average precision, recall and F1-scores of 0.67, 0.61 and 0.635 respectively.
BibTeX
@inproceedings{vijayaraghavan2016deepstance,
  author = {Prashanth Vijayaraghavan and Ivan Sysoev and Soroush Vosoughi and Deb Roy},
  title = {{DeepStance at SemEval-2016 Task 6: Detecting Stance in Tweets Using Character and Word-Level CNNs}},
  booktitle = {SemEval @ NAACL},
  year = {2016},
  doi = {10.18653/v1/s16-1067},
}
Figure