SINAI at SemEval-2021 Task 5: Combining Embeddings in a BiLSTM-CRF model for Toxic Spans Detection

Flor Miriam Plaza-del-Arco, Pilar López-Úbeda, L. Alfonso Ureña-López, M. Teresa Martín-ValdiviaACL 2021

TL;DR

This paper tackles the crucial task of identifying specific "toxic spans" within online text, going beyond simply classifying an entire message as toxic. The SINAI team developed a BiLSTM-CRF model that leverages combinations of pre-trained word embeddings (GloVe, BERT, and XLM) to achieve strong performance, ranking 29th out of 91 teams in the SemEval-2021 competition. This work highlights the importance of pinpointing why text is toxic for effective moderation.

What Problem Does This Paper Address?

The proliferation of online communication has unfortunately led to an increase in toxic and offensive language. While many NLP efforts have focused on classifying entire texts as offensive or not, this binary approach falls short. Identifying the specific words or phrases (spans) that contribute to toxicity is vital for effective moderation. This allows for a deeper understanding of why a text is toxic, enabling human moderators to more efficiently pinpoint problematic content on social media platforms and helping to mitigate the negative psychological effects on users. This paper addresses the challenge of accurately detecting these toxic spans.

Methods

The SINAI team employed a deep learning-based approach for Named Entity Recognition (NER) to identify toxic spans. Their core methodology is a Bidirectional Long Short-Term Memory (BiLSTM) network coupled with a Conditional Random Field (CRF) decoding layer, commonly known as a BiLSTM-CRF model. This architecture is well-suited for sequence labeling tasks, as the BiLSTM captures contextual information from both past and future words in a sentence, while the CRF layer uses this context to predict the most likely sequence of tags.

A key aspect of their approach was experimenting with different word embeddings as input to the BiLSTM layer:

  • Static Word Embeddings: They used GloVe embeddings, which assign a fixed vector to each word, irrespective of its context.
  • Contextual Word Embeddings: They explored BERT and XLM embeddings. These are context-dependent, meaning the word representation changes based on the surrounding words, allowing for a more nuanced understanding of meaning.

The researchers specifically tested combinations of these embeddings, hypothesizing that a richer representation would lead to better performance.

Datasets

The study utilized the English dataset provided for SemEval-2020 Task 5: Toxic Spans Detection. This dataset was split into three subsets:

  • Train: 7,939 instances
  • Trial: 690 instances
  • Test: 2,000 instances

Each instance consists of the text and a list of toxic spans, defined by character offsets. The paper also references the SemEval-2021 Task 5 dataset for their participation.

Key Insights and Ideas

  • Beyond Binary Classification: The paper strongly advocates for the necessity of toxic span detection over simple text classification, emphasizing its role in understanding why text is toxic.
  • The Power of Combination: Combining different types of pre-trained word embeddings (static and contextual) significantly improved the model's performance compared to using any single embedding type alone. This suggests that a multi-faceted representation of words is crucial for capturing the nuances of toxic language.
  • BiLSTM-CRF as a Robust Choice: The BiLSTM-CRF architecture proved effective for this task, demonstrating its versatility beyond traditional NER applications.
  • Toxic Spans Detection is Challenging: The relatively low scores across many participants in the competition highlight the inherent difficulty of this task, underscoring the need for continued research.

Key Results

The SINAI team achieved an F1-score of 0.6727 with their best-performing model, which combined BERT, XLM, and GloVe embeddings. This result placed them 29th out of 91 participating teams in the SemEval-2021 Task 5 competition.

Individually, the contextual embeddings performed slightly better than the static one:

  • GloVe: 0.6618
  • BERT: 0.6606
  • XLM: 0.6635

However, the combination of all three embeddings (BERT + XLM + GloVe) yielded the highest F1-score of 0.6727, demonstrating the benefit of their ensemble approach.

Limitations and Future Work

The authors acknowledge that the task of toxic spans detection is challenging and requires further research. They suggest that future work could explore incorporating other linguistic features, such as emotions, which are closely linked to offensive language. The paper also notes that their system achieved a respectable rank but was outperformed by the top team (HITSZ-HLT) by 3.56%, indicating room for improvement.

Summary updated

BibTeX
@inproceedings{plazadelarco2021sinai,
  author = {Flor Miriam Plaza-del-Arco and Pilar López-Úbeda and L. Alfonso Ureña-López and M. Teresa Martín-Valdivia},
  title = {{SINAI at SemEval-2021 Task 5: Combining Embeddings in a BiLSTM-CRF model for Toxic Spans Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure