TL;DR
This paper tackles the challenge of automatically identifying whether participants in online debates agree or disagree with the main topic. They propose a novel three-step method that first classifies agreement/disagreement between individual posts (local) and then aggregates these local judgments to determine a participant's overall stance (global), outperforming methods that directly classify global positions.
What Problem Does This Paper Address?
Online debate forums are rich sources of public opinion, but understanding these opinions automatically is difficult due to informal language and dynamic conversation structures. A key challenge is discerning a participant's overall stance on a debate's main topic (global position) from their individual posts. This is complicated because a post might not directly address the main topic but rather respond to a previous post, creating a "local" agreement or disagreement. Existing methods often struggle to bridge this gap between local interactions and global stances, leading to less accurate classification of overall opinions.
Methods
The authors propose a three-step approach:
- Local Classification: They build a classifier to determine if a pair of posts agree or disagree. This involves identifying explicit (quoted) or implicit (reply to the previous post) links between posts.
- Aggregation: They aggregate these local post-pair classifications over all interactions between two participants to determine their local agreement/disagreement with each other. The intuition here is that participants tend to maintain consistent stances throughout a discussion.
- Global Classification: Finally, they infer a participant's global position (agree/disagree with the seed post) by traversing the thread structure from the seed post. They use a "walking the tree" approach, prioritizing the shortest path from the seed post to a user to determine their stance.
Key to their local classifier is the exploration of various features:
- Words: Unigrams and bigrams, focusing on adjectives, verbs, and adverbs.
- Sentiment: Using SentiWordNet to derive positive and negative scores for words in a post.
- Emotional: Features like emoticons, capital letters, foul words, and punctuation (exclamation/question marks) to capture emotional intensity.
- Durational: Length of posts (words/characters) and time differences between posts, inspired by conversation analysis.
They use L2-regularized logistic regression for local classification due to its good performance and ability to estimate class probabilities, which are crucial for aggregation.
Datasets
The authors evaluated their method on two real-world datasets:
- US Message Board (usmb.com): A general discussion forum.
- Political Forum (politicalforum.com): A forum focused on political discussions.
These datasets were crawled and then manually annotated. Seed posts were not annotated or classified. Annotations were made for both global (with respect to the seed post) and local (with respect to the immediate parent post) positions. The ratio of "agree" to "disagree" posts was approximately 2:1.
Key Insights and Ideas
- Bridging Local and Global: The core insight is that aggregating local agreement/disagreement judgments between posts and then between users is more effective for determining global stances than directly classifying global positions.
- The Power of Aggregation: Aggregating local classifications helps "wash out" errors made in individual local judgments, leading to more robust global stance detection. This is particularly beneficial when local opinions are not clearly expressed.
- Feature Importance: Sentiment, emotional, and durational features significantly improve local classification performance beyond just using word features. Durational features, in particular, showed substantial improvement.
- Logistic Regression's Suitability: L2-regularized logistic regression proved effective for both local classification and for aggregating probabilities to estimate local positions between users.
Key Results
- Local Classification: Logistic regression with all features (words, sentiment, emotional, durational) achieved the highest accuracy and F-measure on both datasets, outperforming Naive Bayes and SVMs. For example, on the 'pf' dataset, logistic regression achieved 0.68 accuracy and 0.77 F-measure.
- Global Classification: The proposed three-stage method ("with aggregation") consistently outperformed "without aggregation" baselines across all classifiers and datasets.
- On the 'usmb' dataset, logistic regression with aggregation achieved 0.64 accuracy and 0.77 F-measure, compared to 0.60 accuracy and 0.63 F-measure without aggregation.
- On the 'pf' dataset, logistic regression with aggregation achieved 0.68 accuracy and 0.76 F-measure, compared to 0.65 accuracy and 0.77 F-measure without aggregation.
- The improvement from aggregation was more pronounced on the 'usmb' dataset, likely due to a higher average number of post exchanges per user pair.
Limitations and Future Work
- Neutral Class: The current framework only classifies positions into "agree" and "disagree." The authors acknowledge that a "neutral" class exists where users express no clear stance, and plan to extend their framework to handle this.
- Off-Topic/Topic Drift: The current method assumes posts are relevant to the main topic. Future work will explore how to handle off-topic or topic-drift posts, which are common in long discussions and increase classification difficulty.
- Other Predictive Factors: The authors intend to investigate other factors that might influence agreement/disagreement in online debates to further improve classification performance.
Summary updated