Classification Of Spanish Election Tweets (COSET) 2017 : Classifying Tweets Using Character and Word Level Features

Ankush Khandelwal, Sahil Swami, Syed Sarfaraz Akhtar, Manish ShrivastavaIberEval@SEPLN 2017 · OpenReview

Introduction

Social media platforms, particularly Twitter, have become essential arenas for political discourse and public sentiment. However, the high volume and informal nature of these communications—characterized by abbreviations, slang, and misspellings—present significant challenges for automated analysis. This paper describes a machine learning pipeline developed for the Classification Of Spanish Election Tweets (COSET) 2017 shared task. The objective was to categorize tweets related to the 2015 Spanish General Election into five distinct classes: Political issues, Policy issues, Personal issues, Campaign issues, and Other issues.

Methodology

The authors approached the task as a supervised text classification problem, implementing a pipeline that emphasizes robust feature engineering to overcome the noise inherent in social media text.

Preprocessing and Hashtag Decomposition

Standard preprocessing steps included tokenization, lowercasing, and stemming using the Snowball algorithm. URLs and user mentions were replaced with generic placeholders. A key innovation in this pipeline was hashtag decomposition. By splitting camel-cased or underscored hashtags (e.g., #CarlosBarraGalán) into their constituent words ("Carlos", "Barra", "Galan"), the model could extract semantic value from tokens that are often treated as noise or out-of-vocabulary terms.

Feature Engineering

The system utilized a high-dimensional composite feature vector:

  • Character N-grams: Filtered for a minimum frequency of 10 to capture morphological patterns.
  • Word N-grams: Unigrams to trigrams, also frequency-filtered.
  • Reference Tokens: Specific tokens highly correlated with a single class (appearing >60% in one class and at least 5 times overall).
  • Word Embeddings: Pre-trained Spanish Word2Vec embeddings were used to represent tweets as the mean of their constituent word vectors, bridging the semantic gap between synonymous terms.

Model Selection

The study compared three supervised learning algorithms: Support Vector Machines (SVM) with a Radial Basis Function (RBF) kernel, Random Forests, and Extra Trees. Hyperparameters were optimized using grid search.

Dataset Analysis

The system was evaluated using the COSET 2017 corpus. The task was divided into two phases: a supervised learning phase using a small annotated dataset and a large-scale inference phase on 16 million unlabelled tweets.

The distribution of tweet categories across the Training and Development sets in the COSET 2017 corpus.

As shown in the table above, the dataset exhibits class imbalance, with "Policy" and "Political" issues being the most prevalent, while "Campaign" issues were significantly underrepresented in the development set.

Results and Discussion

Experimental results highlighted the superiority of character-level features and the effectiveness of the SVM classifier. On the Phase 1 test set, the Kernel SVM achieved an F1-macro score of 0.6054. When applied to the massive Phase 2 corpus, the model's performance scaled remarkably well, reaching an F1-macro score of 0.8509.

Performance comparison of various features across different machine learning models on the development set.

Key Insights

  • Character vs. Word Features: Character n-grams consistently outperformed word n-grams across all models. For instance, using SVM, character n-grams achieved 60.8% accuracy compared to 52.8% for word n-grams. This confirms that character-level modeling is more resilient to the informal grammar and misspellings typical of Twitter.
  • Feature Fusion: The highest development accuracy (69.8%) was achieved by combining all features, including embeddings and reference tokens. This suggests that dense semantic representations and sparse lexical features provide complementary information.
  • Model Robustness: Tree-based models like Random Forest struggled with the small training set, often overfitting the data and resulting in lower test performance (~0.44 F1) compared to the more stable SVM.

Conclusion and Future Work

This research demonstrates that a combination of character-level features, hashtag decomposition, and word embeddings can effectively classify noisy political tweets. While the SVM model proved robust, the primary limitation remains the scarcity of manually annotated training data. Future work will explore the integration of richer linguistic features—such as Part-of-Speech tagging and Named Entity Recognition—and the application of deep learning architectures to further improve classification accuracy in the dynamic landscape of social media discourse.

Summary updated

BibTeX
@inproceedings{khandelwal2017classification,
  author = {Ankush Khandelwal and Sahil Swami and Syed Sarfaraz Akhtar and Manish Shrivastava},
  title = {{Classification Of Spanish Election Tweets (COSET) 2017 : Classifying Tweets Using Character and Word Level Features}},
  booktitle = {IberEval@SEPLN},
  year = {2017},
  url = {https://openreview.net/forum?id=_Yda9AwonUw},
}
Figure