Social Media in State Politics

Lei Qi, Rihui Li, Johnny Wong, Wallapak Tavanapong, David A. Peterson2017 · DOI

TL;DR

This paper tackles the challenge of automatically classifying tweets from state legislators into predefined policy agenda topics, a crucial step for understanding political discourse. They introduce a novel data augmentation technique and demonstrate that Convolutional Neural Networks (CNNs) significantly outperform traditional methods like SVMs and LSTMs for this task, achieving high accuracy in predicting key policy themes.

What Problem Does This Paper Address?

Understanding how politicians use social media, particularly Twitter, to shape and communicate policy agendas is a growing area of interest. However, manually analyzing the sheer volume of tweets from state legislators is impractical. Existing methods for classifying political discourse often focus on national-level policies or are not well-suited for the brevity and informal nature of tweets. This paper aims to bridge this gap by developing an automated system to categorize state legislators' tweets into the established Policy Agendas Project (PAP) framework, enabling researchers to systematically track policy discussions at the state level.

Methods

The researchers explored three machine learning algorithms for tweet classification: Support Vector Machine (SVM), Convolutional Neural Networks (CNN), and Long Short-Term Memory Network (LSTM). They also introduced a novel synthetic data augmentation method that leverages word embeddings. This technique works by selecting an "anchor" word in a tweet and replacing it with its closest synonyms from a pre-trained word embedding space, generating new training samples. This is particularly useful for addressing the class imbalance problem common in such datasets. For feature extraction, they experimented with TF-IDF on character and word n-grams, with and without stop word removal.

Datasets

The study utilized a dataset of approximately 21,336 tweets collected from 472 Twitter accounts of state legislators across eleven states. For their experiments, they focused on tweets from Iowa and Nebraska. These tweets were manually labeled by political science students according to the PAP codebook, creating a benchmark dataset for evaluating their classification models.

Key Insights and Ideas

  • CNNs Excel at Tweet Classification: The paper highlights that CNNs, particularly when adapted for text classification by learning features automatically, significantly outperform traditional methods like SVM and LSTM for this specific task.
  • Data Augmentation is Crucial for Imbalance: The proposed synthetic data augmentation method, which generates new training samples based on word embeddings, proved effective in improving classification performance, especially for imbalanced datasets. This suggests that simply having more data isn't enough; the quality and diversity of augmented data matter.
  • Character-Level Features are Robust: For informal text like tweets, character-level n-grams showed better accuracy than word-level grams, likely due to their ability to tolerate typos and variations common in social media.
  • Predicting Top Topics is Feasible: The research demonstrates that their CNN model can reliably predict the top three most discussed policy agenda topics in a given state and month, offering valuable insights for political science research.

Key Results

  • The CNN model achieved the best F1 score of 78.3%, outperforming SVM by approximately 10% and LSTM by about 7%.
  • The proposed data augmentation method improved classification accuracy by around 2%.
  • For predicting the top 3 policy agenda topics, the model achieved a rough match accuracy of 0.91 for Iowa and 0.91 for Nebraska, meaning it correctly identified the relevant topics most of the time. The exact match accuracy was 0.82 for Iowa and 0.75 for Nebraska, indicating that while the correct topics were often identified, their precise order wasn't always perfect.
  • The top 1 policy topic was consistently predicted correctly across all months and states.

Limitations and Future Work

The authors acknowledge that while their CNN model provides good prediction accuracy for the top policy topics, there's still room for improvement, particularly in precisely ordering the top 3 topics. They also note that some tweets are inherently ambiguous, even for human experts, making perfect classification challenging.

Future work includes investigating active deep learning methods to streamline the process of obtaining ground truth labels by recommending the minimum number of tweets domain experts need to label for high prediction accuracy. They also plan to explore how their model can be used to track the diffusion of policy agendas across states and compare state-level agendas with national ones.

Summary updated

Abstract

Twitter is a popular online microblogging service that has become widely used by politicians to communicate with their constituents. Gaining understanding of the influence of Twitter in state politics in the United States cannot be achieved without proper computational tools. We present the first attempt to automatically classify tweets of state legislatures (policy makers at the state level) into major policy agenda topics defined by Policy Agendas Project (PAP), which was initiated to group national policies. We investigated the effectiveness of three popular machine learning algorithms, Support Vector Machine (SVM), Convolutional Neural Networks (CNN), and Long Short-Term Memory Network (LSTM). We proposed a new synthetic data augmentation method to further improve classification performance. Our experimental results show that CNN provides the best F1 score of 78.3%. The new data augmentation method improves the classification perfromance by about 2%. Our tool provides a good prediction of the top three popular PAP topics in each month, which is useful for tracking popular PAP topics over time and across states and for comparing with national policy agendas.
BibTeX
@misc{qi2017social,
  author = {Lei Qi and Rihui Li and Johnny Wong and Wallapak Tavanapong and David A. Peterson},
  title = {{Social Media in State Politics}},
  year = {2017},
  doi = {10.1145/3110025.3110097},
}
Figure