TL;DR
This paper tackles the challenge of limited labeled data in detecting online sexism by employing a semi-supervised learning approach. They introduce a novel strategy of augmenting labeled data using synonym replacement based on TF-IDF, combined with consistency training, to significantly improve model performance.
What Problem Does This Paper Address?
Online sexism is a pervasive issue, and accurately detecting it is crucial for creating safer online spaces. However, building high-performing models for such tasks is often hampered by a scarcity of labeled data. Traditional supervised learning methods struggle to generalize well with limited examples. Semi-supervised learning offers a promising avenue by leveraging large amounts of unlabeled data, but ensuring the unlabeled data aligns with the specific domain of the task (like online sexism) can be difficult. This paper addresses this by proposing a method to effectively utilize labeled data for both supervised training and for generating augmented data that maintains domain relevance for consistency training.
Methods
The core of their approach is a consistency training framework. This framework encourages the model to produce similar predictions for slightly perturbed versions of the same input. They combine this with a data augmentation strategy. Instead of augmenting unlabeled data (which might be out-of-domain), they focus on augmenting the existing labeled data.
Their specific data augmentation technique involves synonym replacement based on TF-IDF. Here's how it works:
- TF-IDF Calculation: They first calculate TF-IDF (Term Frequency-Inverse Document Frequency) scores for words within the labeled dataset. TF-IDF helps identify words that are important within a specific document (sentence, in this case) but not overly common across the entire corpus.
- Synonym Replacement: Words with TF-IDF scores below a certain threshold are identified. For these words, they find synonyms and randomly replace the original word with one of its synonyms. This process introduces variations while aiming to preserve the original sentence's meaning.
The model itself is based on a BERT-base architecture with three linear layers. The training objective is a combination of supervised loss (using both cross-entropy and Focal Loss) and unsupervised loss (KL divergence for consistency). They found that using Focal Loss for the supervised component further boosted performance.
Datasets
- Training Data: The paper mentions using 14,000 entries in English for training, with 3,398 identified as sexist. This data is split into 90% for training, 5% for validation, and 5% for testing.
- Unsupervised Learning Data: For the unsupervised learning setting, they generated augmented data from the training set, resulting in another 14,000 examples.
- External Data (Mentioned as problematic): They briefly mention that using unlabeled data from sources like Gab and Reddit for consistency training led to performance degradation, reinforcing their decision to augment labeled data.
Key Insights and Ideas
- Augmenting Labeled Data is Key: The most significant insight is that for domain-specific tasks with limited labeled data, augmenting the labeled data itself is more effective than trying to find and use out-of-domain unlabeled data for consistency training. This is because labeled data is inherently domain-relevant.
- TF-IDF for Meaningful Augmentation: Using TF-IDF to guide synonym replacement ensures that less important words are modified, minimizing the risk of altering the core meaning of the sentence and thus preserving domain relevance.
- Focal Loss Boosts Performance: Incorporating Focal Loss into the supervised learning component, in addition to standard cross-entropy, provided a noticeable improvement in performance.
Key Results
The paper reports a macro F1 score of 0.8180 for their method on SemEval-2023 Task 10 (Explainable Detection of Online Sexism), ranking 57th among all teams. This represents a significant improvement over their baseline model, which achieved a score of 0.8004 using only supervised learning with cross-entropy loss. When using Focal Loss with the baseline, the score was 0.8108.
Their experiments also demonstrated that using unlabeled data from Gab and Reddit for consistency training resulted in lower scores (0.7855 for CE, 0.7933 for FL) compared to using their augmented labeled data. They also found that an optimal threshold of 0.8 for TF-IDF in their augmentation strategy yielded the best macro F1 score of 0.818.
Limitations and Future Work
- Threshold Selection: The authors acknowledge that the selection of the TF-IDF threshold for synonym replacement is a crucial hyperparameter that significantly impacts performance. They found an optimal threshold through experimentation but suggest further research into more robust methods for determining this value.
- Data Augmentation Strategies: They plan to explore more diverse and high-quality data augmentation strategies to generate better samples for consistency training.
- Loss Functions and Training Skills: Future work will also focus on designing improved loss functions specifically for consistency training and investigating advanced training techniques.
Summary updated