Abstract
This paper describes our approach for the Detecting Stance in Tweets task (SemEval-2016 Task 6). We utilized recent advances in short text categorization using deep learning to create word-level and character-level models. The choice between word-level and character-level models in each particular case was informed through validation performance. Our final system is a combination of classifiers using word-level or character-level models. We also employed novel data augmentation techniques to expand and diversify our training dataset, thus making our system more robust. Our system achieved a macro-average precision, recall and F1-scores of 0.67, 0.61 and 0.635 respectively.
Introduction
Stance detection is the task of determining an author's position—typically "Favor," "Against," or "None"—toward a specific target. While often confused with sentiment analysis, stance detection is uniquely target-specific. For instance, a positive sentiment toward one political figure often implies a negative stance toward their opponent. Detecting these nuances in tweets is challenging due to the short, noisy nature of the text, which frequently includes slang, typos, and idiosyncratic hashtags.
Methodology
Our system, DeepStance, employs two primary deep learning architectures to capture different linguistic granularities:
- Character-Level CNN (CharCNN): This model processes text character-by-character, making it inherently robust to misspellings and the informal language prevalent on Twitter.
- Word-Embedding CNN: This model utilizes word embeddings to map tokens into a high-dimensional vector space, capturing semantic relationships more effectively on smaller datasets.
Data Augmentation: The "Secret Sauce"
Given the limited size of the official SemEval-2016 dataset, we implemented two novel data augmentation strategies:
- Hashtag Mining: We used Mutual Information to identify hashtags strongly correlated with specific stances. These hashtags served as proxies to collect 1.7 million additional tweets from historical archives.
- Word2Vec Substitution: We generated 500,000 synthetic tweets per target by replacing words with contextually similar synonyms identified via Word2Vec embeddings. This approach provided better coverage for informal language than traditional dictionary-based methods.
Results and Discussion
DeepStance achieved a macro-average F1-score of 0.635, ranking 8th out of 19 teams overall. Notably, the system secured the #1 rank for the "Legalization of Abortion" target.
A key finding was that the optimal model architecture is target-dependent. For example, character-level models showed significant improvement for the "Hillary Clinton" target when augmented with sufficient data, whereas word-level models were often more efficient for targets with less available training data.
Conclusion
The DeepStance system demonstrates that combining character and word-level features with aggressive data augmentation can effectively address the challenges of stance detection in noisy social media text. Future work will explore unsupervised pre-training and more sophisticated "bootstrapping" techniques to further refine the quality of augmented training samples.
Summary updated