TL;DR
This paper tackles the challenge of automatically identifying controversial comments in online discussions, crucial for improving user interaction and conflict resolution. They propose a novel multi-task Convolutional Neural Network (CNN) that leverages incidentally supervised signals (like vote ratios and article genres) to predict controversy without manual labeling, achieving strong performance.
What Problem Does This Paper Address?
Online discussion forums often get bogged down in repetitive arguments, hindering productive dialogue and consensus building. Identifying "controversial" comments – those that spark inter-group agreement conflicts – is key to guiding users towards more constructive interactions, as demonstrated by previous work. However, existing methods rely on manual labeling of controversy, which is time-consuming, expensive, and cannot keep pace with the rapid evolution of online discussions across diverse domains. This paper aims to automate controversy prediction at scale by developing a system that can learn to identify controversial comments using readily available, incidentally supervised signals, rather than requiring explicit human annotation.
Methods
The core of the proposed solution is a multi-task Convolutional Neural Network (CNN). The authors build upon a single-task CNN (ST) designed for controversy prediction and extend it into a multi-task architecture (MT). The MT model incorporates a "genre-encoder" that simultaneously predicts the article's genre. This genre information is then fed into the controversy prediction task, allowing the model to learn genre-specific controversy patterns.
The "incidental supervision" is a key innovation. Instead of relying on human-labeled controversy, they use comment vote ratios: comments with a conflicted agreement (i.e., not a clear 2/3 majority of up or down votes) are labeled as controversial. Similarly, article genres are used as an incidental signal for structure.
The paper also explores various CNN module modifications, including different convolution methods, activation schemes, and pooling mechanisms, to optimize performance. They also perform ablation studies to understand the impact of different feature types (sentiment, discourse, topic) on controversy prediction.
Datasets
The authors collected a corpus of comments from the Austrian newspaper DerStandard.at in 2015. This corpus includes:
- 813k comments.
- 8.9k controversial comments and 12.6k non-controversial comments (after filtering for duplicates and short comments).
- Each comment is associated with an article genre tag.
- User comments have up and down votes, which are used to derive the incidental controversy signal.
The corpus is preprocessed to remove noise, create special tokens for discourse and reactionary sentiment (emoticons), and separate compound words. Word embeddings were pre-trained on a large German text corpus.
Key Insights and Ideas
- Incidental Supervision is Powerful: The paper demonstrates that using readily available signals like vote ratios and article genres can effectively train a controversy prediction model, bypassing the need for costly manual labeling.
- Multi-task Learning Benefits: Jointly predicting controversy and article genre improves controversy prediction performance, suggesting that genre context is a valuable signal.
- Feature Importance: Topic (nouns) emerged as the most crucial feature for controversy prediction, followed by discourse markers and sentiment.
- Context Matters: The model learns to interpret the influence of individual words within their context, distinguishing between words that increase or decrease controversy.
- Genre-Specific Controversy: Different genres have distinct patterns of controversy, with political terms driving controversy in "domestic politics" and war-related terms in "international politics."
Key Results
- The proposed multi-task CNN (MT) models consistently outperformed baseline methods like Multinomial Naive Bayes (MNB), Logistic Regression (LR), and FastText.
- The MT models achieved performance improvements of 2-4 percentage points over the single-task CNN (ST) model, even with fewer parameters.
- Ablation studies revealed that topic-related features (noun removal) had the largest negative impact on controversy prediction when removed, indicating their high importance. Discourse markers were also significant, followed by sentiment features.
- The best-performing model, MT+PNC+SPool+Swish, significantly improved AUCROC and accuracy over simpler variants.
Limitations and Future Work
- Language Independence: While the text normalization is language-agnostic, the pre-trained word embeddings are specific to German. Future work could explore language-independent embeddings.
- Controversy Drift: The current work focuses on a static snapshot of controversy. The authors suggest extending data collection to study how controversy evolves over time.
- Domain Specificity: The current model is trained on news discussions. Its performance on other online discussion platforms (e.g., social media) might differ and warrants further investigation.
Summary updated