shefnlp at SemEval-2023 Task 10: Compute-Efficient Category Adapters

Thomas Pickard, Tyler Loakman, Mugdha PandyaACL 2023

TL;DR

This paper tackles the challenge of detecting and classifying online sexism with fine-grained categories, crucial for understanding and combating its prevalence. The authors propose a compute-efficient approach using adapters, lightweight modules that adapt pre-trained language models, achieving competitive results on the SemEval-2023 Task 10 across three subtasks.

What Problem Does This Paper Address?

The internet, particularly social media, has become a breeding ground for hate speech and negative sentiment, with online sexism being a significant concern. While detecting general sexism is important, a more nuanced understanding requires fine-grained classification of how sexism is expressed. SemEval-2023 Task 10 specifically addresses this by focusing on classifying sexism in short texts from platforms like Reddit and Gab, with three subtasks increasing in granularity: binary detection (sexist/not sexist), a 4-way classification of sexist content types (threats, derogation, animosity, prejudiced discussions), and an 11-class taxonomy further breaking down these categories. The challenge lies in developing models that can accurately classify these subtle distinctions, especially given the often imbalanced nature of the data and the computational cost of training large language models.

Methods

The core of the authors' approach revolves around adapters, which are small, modular components inserted into pre-trained transformer language models (like RoBERTa-base). Instead of fine-tuning the entire large model, only these adapters are trained, significantly reducing computational cost and parameter count. This makes the models more portable and efficient.

Key methodological contributions include:

  • Adapter Fusion: Combining multiple adapters, trained for different purposes (e.g., Task A adapters fed into Task B adapters), to leverage learned representations.
  • Class-Specific Adapters: Training individual adapters for each specific subtask category, which are then fused for the final classification.
  • Weighted Loss Function: To address class imbalance, the loss function is weighted inversely to the frequency of each class in the training data.
  • Pseudo-labelling: Augmenting the limited labeled data by using a pre-trained model (DistilBERT) to label a large amount of unlabeled data, then incorporating a subset of these pseudo-labeled examples into the training set.
  • Hyperparameter Tuning: Optimizing parameters like learning rate, which they found to be crucial for adapter performance, often benefiting from higher rates than standard fine-tuning.

Datasets

The primary dataset used is from SemEval-2023 Task 10, consisting of approximately 14,000 labeled posts from Reddit and Gab. This data is divided into training, development, and test sets. Additionally, the organizers provided a development set of 2,000 unlabeled items and a large corpus of 1 million unlabeled posts from Reddit and Gab, which were utilized for continued model pre-training and pseudo-labelling.

Key Insights and Ideas

  • Adapters are Compute-Efficient and Effective: The paper strongly demonstrates that adapters can achieve performance comparable to full model fine-tuning but with significantly fewer trainable parameters, making them a practical choice for resource-constrained environments or multi-task learning.
  • Hierarchical Learning Potential: The "adapter stack" approach, where adapters trained for a simpler task (Task A) are used as input for adapters trained on a more complex task (Task B), shows promise for hierarchical classification problems.
  • Weighted Loss and Pseudo-labelling are Crucial for Imbalanced Data: For highly imbalanced datasets like Task C, combining weighted loss with pseudo-labelling is essential to improve performance.
  • Hyperparameter Sensitivity: Adapter training can be sensitive to hyperparameters, particularly learning rate, which can be significantly higher than for full fine-tuning, leading to better results.
  • Dataset Specificity Matters: Observations suggest that model performance can vary between the competition's development set and the final test set, highlighting the importance of careful evaluation and potential domain shifts.

Key Results

The authors achieved competitive results across the three subtasks:

  • Task A (Binary Detection): Their best system achieved an F1 score of 0.8181, ranking 56th out of 84 submissions. This was comparable to a fully fine-tuned baseline but with greater efficiency.
  • Task B (4-way Classification): Their system achieved an F1 score of 0.5890, ranking 43rd out of 69 submissions.
  • Task C (11-class Classification): Their system achieved an F1 score of 0.3811, ranking 37th out of 63 submissions.

Limitations and Future Work

  • Adapter Stacking Performance: While promising, the adapter stacking approach did not yield the highest performance on the competition development set. The authors suggest further refinement for hierarchical classification problems.
  • Inter-Annotator Disagreement: The paper acknowledges the criticism of using "gold standard" labels for subjective tasks like sexism detection, where annotator opinions can differ. They plan to explore the effect of inter-annotator disagreement on bias in sexism detection in future work.
  • Limited Hyperparameter Optimization for Tasks B & C: Due to time constraints, hyperparameters were primarily optimized for Task A, and the same configuration was applied to Tasks B and C. Separate optimization could potentially yield better results.

Summary updated

BibTeX
@inproceedings{pickard2023shefnlp,
  author = {Thomas Pickard and Tyler Loakman and Mugdha Pandya},
  title = {{shefnlp at SemEval-2023 Task 10: Compute-Efficient Category Adapters}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure