TL;DR
This paper introduces a novel approach to detecting toxic online comments by incorporating sentiment analysis, arguing that while toxic keywords can be easily disguised, the underlying emotional tone is harder to mask. They demonstrate that sentiment information significantly improves toxicity detection, especially when users attempt to subvert detection systems by altering keywords.
What Problem Does This Paper Address?
Online communities are plagued by toxic content, including cyberbullying, racism, and hate speech. Current moderation strategies, relying on automated keyword filters or human oversight, struggle to keep pace with the sheer volume of online communication and the evolving tactics of malicious users. Toxic users can easily circumvent keyword-based filters by making minor changes to words (e.g., misspellings) or using synonyms, a practice termed "subversion." This paper addresses the challenge of detecting toxicity, particularly in the face of such subversive tactics, by exploring the underutilized dimension of sentiment analysis.
Methods
The authors propose a two-pronged approach: first, building a robust sentiment detection tool, and second, integrating this tool into a toxicity detection system.
For sentiment detection, they experiment with combining six different sentiment lexicons (SentiWordNet, Afinn, Bing Liu, General Inquirer, Subjectivity Clues, and NRC). They develop preprocessing steps to handle negations (using a fixed window approach for efficiency) and identify sentiment-carrying idioms. They also explore different strategies for aggregating word sentiments: using all words versus only the top-scoring words, and optionally incorporating negation and word frequency modifiers.
For toxicity detection, they employ a deep neural network (specifically a bi-GRU layer followed by a linear classifier), inspired by successful models in the Kaggle toxicity competition. This network takes both word embeddings (from fastText) and character embeddings as input. Crucially, they integrate sentiment scores derived from their sentiment tool as additional input features for the neural network. To simulate subversive users, they create a substitution list to replace common toxic keywords with harmless alternatives, introducing noise into the toxic messages.
Datasets
The paper utilizes three publicly available datasets for evaluating their toxicity detection system:
- Reddit dataset: A large corpus of over 880,000 comments from various subreddits, annotated by the Community Sift tool. For this study, comments were categorized based on bullying, fighting, sexting, vulgarity, and racism.
- Wikipedia Talk Labels dataset: Over 100,000 comments from English Wikipedia talk pages, manually annotated by multiple Crowdflower workers.
- Kaggle toxicity competition dataset: Approximately 160,000 comments from English Wikipedia talk pages, annotated with six binary labels (toxic, severe_toxic, obscene, threat, insult, and identity_hate), allowing for a seven-level toxicity scale.
Key Insights and Ideas
- Sentiment as a Subversion Countermeasure: The core idea is that while users can easily change words to evade filters, altering the underlying emotional tone of a message is significantly harder. This makes sentiment a powerful signal for detecting toxicity, especially when keywords are masked.
- Lexicon Combination is Key: No single sentiment lexicon is perfect. Combining multiple lexicons, particularly those that accurately capture negative sentiments (SentiWordNet, Afinn, Bing Liu), and averaging their scores, yields better results than any individual lexicon.
- "Top Words" vs. "All Words" for Sentiment: For sentiment analysis, focusing on the "top words" with the strongest sentiment scores can be more effective than summing all word sentiments, especially in short messages where a few strong sentiments can outweigh many weak ones.
- Negations and Frequency Modifiers Offer Minor Gains: While incorporating negation handling and word frequency modifiers can slightly improve sentiment detection, their impact is less pronounced than the choice of lexicons or the "top words" strategy.
- Subversion Significantly Degrades Keyword-Based Detection: When toxic keywords are replaced with harmless ones, the accuracy of toxicity detection drops considerably, primarily due to a loss in recall.
- Sentiment Recovers Lost Accuracy: Integrating sentiment information into the toxicity detection model significantly mitigates the accuracy loss caused by subversion, demonstrating its robustness against keyword manipulation.
Key Results
- Sentiment-Toxicity Correlation: A clear negative correlation was observed between sentiment scores and toxicity scores across all three datasets, confirming the intuition that toxic messages tend to have negative sentiment.
- Sentiment Improves Toxicity Detection: Incorporating sentiment information into the deep neural network model statistically significantly improved toxicity detection accuracy in most cases.
- Impact of Subversion: Subversion led to a substantial drop in accuracy (e.g., from 93.2% to 77.2% on the Kaggle dataset without sentiment).
- Sentiment Mitigates Subversion Impact: With subversion, adding sentiment information improved accuracy by up to 3% (on the Kaggle dataset), with statistically significant p-values in most cases.
- Robustness: The proposed system, even without sentiment, performed well, likely due to the robustness of character embeddings against misspellings and the inherent noisiness of user-generated content.
Limitations and Future Work
- Lexicon Limitations: The paper acknowledges that existing sentiment lexicons are often designed for longer, more traditional documents and may miss insults or politically incorrect language commonly found in online conversations. This can lead to misclassifications, where messages using words like "gay" as an insult are incorrectly labeled as positive.
- Single-Line Messages: The current work focuses on single-line messages. Future work could extend this to handle entire conversations, allowing for the incorporation of contextual information and the detection of sentiment shifts that signal disruptive toxic comments.
- Specific Toxicity Types: The current focus is on negative toxicity. The authors suggest that other forms of toxicity, like fraud or sexual grooming, might involve more positive sentiments, and differentiating these types could further enhance sentiment-based detection.
- Subversion Simulation: The subversion simulation used a fixed substitution list. Future work could explore more dynamic or sophisticated subversion techniques.
Summary updated