TL;DR
This paper introduces "Orthrus," an ensemble approach for identifying toxic "spans" (specific phrases) in online text. By combining a lexicon-based method with a supervised learning model, Orthrus achieves a significant improvement in F1 score, demonstrating the power of hybrid approaches for nuanced toxicity detection. The study also explores the correlation between emotions and toxicity, suggesting a promising avenue for future research.
What Problem Does This Paper Address?
Online platforms are grappling with the pervasive issue of toxic language, which ranges from hate speech and cyberbullying to general harassment. While many existing tools focus on classifying entire posts as toxic, this paper addresses a more granular problem: identifying the precise "spans" of text that constitute toxicity. This fine-grained detection is crucial for effective moderation and for understanding why a piece of text is considered toxic. The authors highlight that simply relying on word lists can be insufficient, as toxicity can be disguised through subtle word changes, while emotions expressed in text might be harder to conceal and could serve as a more robust indicator.
Methods
The core of the Orthrus system is an ensemble approach that combines two distinct methods:
- Lexicon-Based Approach: This method utilizes a custom-built lexicon, named the "Orthrus lexicon," which was initially compiled from toxic words identified in the SemEval 2021 Task 5 training data. This lexicon was further augmented by incorporating words from other publicly available toxic word lists, with a focus on those that maintained high performance. The authors also experimented with combining their custom lexicon with other established lexicons, evaluating their impact on F1 score.
- Supervised Learning Approach: To capture more nuanced toxicity, the authors employed a supervised learning model using the spaCy library. They experimented with both the
en_core_web_sm(small) anden_core_web_lg(large) English models. The choice of spaCy was influenced by its use in a provided solution script for the task.
The ensemble combines these two approaches in two ways: a union (where a span is considered toxic if either method identifies it) and an intersection (where a span is considered toxic only if both methods agree).
Datasets
The primary dataset used for this research is derived from the Civil Comments dataset. For SemEval 2021 Task 5, a subset of approximately 30,000 toxic posts was selected, and from this, a random sample of 10,000 posts was annotated for toxic spans. This dataset is noted as being particularly valuable due to its consistent labeling for toxic spans, addressing a gap in previous research where datasets often had varying labeling conventions.
Key Insights and Ideas
- Hybrid Approaches Outperform Single Methods: The most significant insight is that combining a lexicon-based approach with a supervised learning model yields superior results compared to either method used in isolation. This highlights the complementary strengths of rule-based and data-driven techniques.
- Emotions as a Proxy for Toxicity: The paper posits that emotions expressed in text might be a more reliable indicator of toxicity than specific toxic words, which can be easily altered. This is a novel angle that could lead to more robust toxicity detection systems.
- The Challenge of Subtle Toxicity: The example of the post "uh, no, he's a belligerent buffoon (and a traitor)" illustrates how toxicity can be conveyed through sentiment and word choice that might not be present in standard profanity lists, underscoring the need for more sophisticated detection methods.
Key Results
- The Orthrus lexicon alone achieved an F1 score of 61.07% on the test dataset.
- The supervised learning approach using spaCy achieved an F1 score of approximately 60% (specifically, 59.61% for the small model and 59.95% for the large model).
- The ensemble approach using the intersection of the lexicon and the spaCy model achieved the highest F1 score of 66.37%.
- The ensemble using the union achieved an F1 score of 61.53%.
- Analysis of emotions on the test dataset indicated that fear, sadness, and anger were the most prevalent emotions, suggesting a correlation with toxicity.
Limitations and Future Work
- Emotion-Toxicity Correlation is Preliminary: While the paper explores the link between emotions and toxicity, the authors state that this study is "still in progress." The current findings suggest a correlation, but further research is needed to solidify this relationship and integrate it effectively into toxicity detection.
- SpaCy Model Performance: The study found that the larger spaCy model (
en_core_web_lg) did not significantly improve performance over the smaller model (en_core_web_sm), despite being slower. This suggests that for this specific task, the smaller, faster model might be preferable. - Future Work: The authors intend to further explore the correlation between emotions and toxicity, aiming to leverage this understanding for improved toxicity detection. They also suggest that the Orthrus lexicon will be made publicly available.
Summary updated