TL;DR
This paper tackles the challenge of pinpointing toxic spans within online text, a crucial step for effective content moderation. The authors propose two novel approaches that leverage pre-trained language models and existing sentence-level toxicity datasets to improve token-level toxicity detection, outperforming a baseline model.
What Problem Does This Paper Address?
The proliferation of social media has led to an explosion of toxic content, making manual moderation an insurmountable task. While automated toxicity detection exists, most systems only classify entire comments as toxic or not. This lacks the granularity needed for effective moderation, as human moderators benefit from knowing which specific parts of a comment are toxic. The core challenge is the scarcity of datasets labeled at the token or span level, making it difficult to train accurate models for this fine-grained detection. This paper aims to address this data scarcity by exploring methods to leverage readily available sentence-level toxicity labels.
Methods
The authors propose two main approaches, both building upon the BERT pre-trained language model:
- Sequential Fine-tuning: This method involves a two-stage fine-tuning process. First, a BERT model is fine-tuned on a large collection of sentence-level toxicity datasets. This "pre-training" on related, data-rich tasks helps the model learn general toxicity patterns. Subsequently, this fine-tuned model is further trained with a token classification head on the actual SemEval-2021 task data.
- Weak Supervision for Augmenting Token Labels: This approach uses sentence-level labels to generate "imperfect" token-level labels. It employs a model loosely based on prior work (Karamanolakis et al., 2019) that takes BERT embeddings and sentence-level predictions to generate attention weights for each token. These weights are used to create a weighted average of token-level predictions, which then informs the generation of new, weakly supervised token labels. These augmented labels are then used to train a token classifier.
Datasets
The paper utilizes several datasets:
- SemEval-2021 Task 5 Data: This is the primary dataset for the toxic spans detection task, providing token-level labels for 9,939 English posts. It was split into training and test sets, with 2,000 samples in the test set.
- HateXplain Dataset: A dataset of 20,148 tweets and Gab posts, labeled as hateful, offensive, or neutral, with rationales for toxic spans.
- Five Sentence-Level Labeled Datasets: These datasets were merged to create a large corpus of sentence-level toxicity labels:
- Jigsaw/Conversation AI toxic comment classification challenge dataset (22,468 toxic samples).
- Hate speech and offensive language dataset (Davidson et al., 2017) (20,620 toxic samples).
- Online harassment dataset (Golbeck et al., 2017) (5,285 harassing samples).
- Impermium dataset for detecting insults in social commentary (1,742 insulting samples).
- OffensEval-2020 subtask-A extended test dataset (3,002 offensive samples).
The combined sentence-level datasets yielded 53,117 toxic samples.
Key Insights and Ideas
- Leveraging Sentence-Level Data is Key: The paper strongly suggests that using readily available sentence-level toxicity data can significantly improve token-level toxicity detection, especially when labeled token data is scarce.
- Sequential Adaptation Outperforms Weak Supervision: The sequential fine-tuning approach, which first trains on sentence-level data and then fine-tunes for token classification, yielded better results than the weak supervision method. This indicates that a more direct transfer of knowledge from sentence to token classification is more effective in this scenario.
- Domain-Specific Knowledge Matters: The improvement from using sentence-level data is more pronounced at higher toxicity thresholds. This suggests that these datasets help models become more adept at identifying highly toxic content.
- Weak Supervision Challenges: The weak supervision approach faced challenges, with a significant portion of generated labels not meeting the >0.5 toxicity score threshold, thus reducing the effective size of the augmented dataset.
Key Results
The paper's best-scoring solution submitted to the contest achieved an F1-Score of 0.6561, ranking 49th out of 91 submissions.
- Baseline Model (BERT only): [email protected] = 0.6224, F1_max = 0.6289
- Weakly Supervised Approach: [email protected] = 0.6644, F1_max = 0.6799 (Improvement of 0.0420 [email protected] over baseline)
- Sequential Adaptation Approach: [email protected] = 0.6678, F1_max = 0.6861 (Improvement of 0.0454 [email protected] over baseline)
The sequential adaptation approach showed a notable improvement of 0.0572 F1 score over the baseline, while the weakly supervised approach improved it by 0.051 F1 score.
Limitations and Future Work
- Weak Supervision Effectiveness: The authors acknowledge that their weak supervision approach did not perform as well as expected, with a large number of generated toxic scores below the 0.5 threshold.
- Training Time: While the weak supervision approach has less training time for the token classification part, the initial sentence classification model training can be time-consuming.
- Future Work: The authors suggest exploring improvements to the weak supervision technique, potentially by using different objective functions for relating token and sentence labels, and investigating multi-task learning.
Summary updated