Vote Prediction on Comments in Social Polls

Isaac Persing, Vincent NgEMNLP 2014 · DOI

Introduction

Predicting individual opinions from social media is a cornerstone of modern sentiment analysis. However, most research focuses on binary stances (e.g., pro-choice vs. pro-life). This paper, presented at EMNLP 2014, introduces the task of "vote prediction" in social polls: determining how a user voted based on the comment they left. Unlike standard sentiment analysis, this task involves mapping noisy, brief, and often sarcastic text to one of several specific poll options.

The Challenge of Social Polls

Social polls on platforms like SodaHead.com allow users to select a predefined answer and provide a justifying comment. These comments present unique NLP challenges due to their casual nature. They are often short, lack explicit sentiment, or require deep contextual knowledge to interpret.

Examples of social poll questions, votes, and comments illustrating the noisy nature of the data.

For instance, as shown in the table above, a comment like "Binders full of women" in response to a presidential debate poll requires knowing the specific political context to map it to a vote for Barack Obama.

Methodology: Beyond Textual Analysis

To address these challenges, the researchers proposed a multi-faceted approach incorporating three types of information:

  1. Textual Features: Standard lemmatized n-grams were augmented with "cue words" (the start of comments), emotion frequencies from the NRC Emotion Word Lexicon, and dependency triples to capture grammatical relationships (e.g., nsubj(inept, obama)).
  2. Demographic Features: User profiles (age, gender, political views, religion) were used as strong priors. Even when partially missing, these features significantly improved prediction accuracy.
  3. Inter-comment Constraints: The paper's primary innovation lies in two types of relational constraints:
  4. QuestionPair: Correlating votes across different but related questions.
  5. VoterPair: Leveraging the agreement history between pairs of users.

To integrate these features at scale, the authors employed a Label Propagation algorithm. This allowed for joint inference across a massive graph of nearly one million comments, a scale that traditional methods like Integer Linear Programming cannot easily handle.

Dataset and Evaluation

The study utilized a custom dataset from SodaHead.com, focusing on the "News & Politics" category. It comprises 997,379 comments across 4,803 questions from 108,462 unique users. This large-scale dataset has been made publicly available to the research community.

Key Results and Insights

The experimental results demonstrate that incorporating social and demographic context is crucial for accurate vote prediction.

Accuracy of various model configurations across different training set sizes for standard vote prediction (a) and arbitrary user vote prediction (b).

As shown in the figures above, the full system (incorporating text, demographics, and both types of constraints) consistently outperforms the baselines. The full system achieved approximately 69.5% accuracy, a significant improvement over the majority vote baseline (~63.5%) and the text-only classifier (~65.5%).

A key insight is the effectiveness of VoterPair constraints. In noisy social media environments, who a user is and who they typically agree with are often more predictive of their vote than the specific words in a short comment. Furthermore, the model proved capable of "Arbitrary User Vote Prediction" (Figure b)—predicting the votes of users who didn't leave a comment at all, based solely on their demographics and voting history.

Limitations and Future Directions

While successful, the approach faces challenges with data sparsity, as many users have incomplete profiles or no voting history ("cold start"). Additionally, the model's performance in less polarized domains (like sports or entertainment) and its ability to handle complex linguistic phenomena like sarcasm remain areas for future exploration.

Summary updated

BibTeX
@inproceedings{persing2014vote,
  author = {Isaac Persing and Vincent Ng},
  title = {{Vote Prediction on Comments in Social Polls}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2014},
  doi = {10.3115/v1/d14-1119},
}
Figure