Sefamerve ARGE at SemEval-2021 Task 5: Toxic Spans Detection Using Segmentation Based 1-D Convolutional Neural Network Model

Selman Delil, Birol Kuyumcu, Cüneyt AksakallıACL 2021

TL;DR

This paper introduces "Waw-unet," a novel 1D convolutional neural network model adapted from U-Net for toxic span detection, treating it as a segmentation problem. While outperforming some transformer models on a specific dataset, it highlights the ongoing challenge of accurately identifying toxic language nuances.

What Problem Does This Paper Address?

The paper tackles the critical problem of toxic span detection, which goes beyond simply classifying an entire document as toxic or not. Instead, it aims to pinpoint the exact words or phrases within a text that contribute to its toxicity. This capability is crucial for developing more sophisticated online content moderation tools, helping to manage semi-automated moderation in platforms like online discussions and news portals. The challenge lies in accurately identifying these granular toxic segments, which can be subtle and context-dependent.

Methods

The authors propose two main approaches:

  • Waw-unet (Segmentation-based): This is their primary contribution. They adapt the U-Net architecture, commonly used for image segmentation, to a 1D convolutional neural network (CNN) for text. The core idea is to treat toxic span detection as a segmentation task where the model outputs a mask indicating toxic characters. They customize the U-Net by adding a "Waw-block," which consists of three convolutional layers with varying kernel sizes (3, 5, and 7) to capture different text patterns. The input is character-based, and the output is a binary mask where '1' signifies a toxic character and '0' otherwise.
  • Transformer-based Models (NER-based): As a baseline and comparison, they also frame toxic span detection as a Named Entity Recognition (NER) problem. Here, toxic spans are treated as a specific named entity. They experiment with pre-trained transformer models like RoBERTa and XLM-RoBERTa.

Datasets

The primary dataset used is the SemEval-2021 Task 5: Toxic Spans Detection dataset. The paper mentions using a standard train/test split of this official release dataset for their experiments. No new benchmarks or datasets were introduced.

Key Insights and Ideas

  • Segmentation as a Powerful Paradigm: Framing toxic span detection as a segmentation problem, rather than just classification or NER, proves to be a viable and effective approach. This allows for more precise localization of toxic content.
  • U-Net Adaptation for Text: Successfully adapting a deep learning architecture (U-Net) primarily designed for image segmentation to the 1D domain of text demonstrates the flexibility of these models.
  • Character-Level Granularity: Using character-based input allows the model to capture subtle toxic patterns that might be missed at a word level, especially in cases of misspellings or creative toxic language.
  • Transformer Models as Strong Baselines: Pre-trained transformer models like RoBERTa and XLM-RoBERTa provide strong performance, highlighting their generalizability and effectiveness in various NLP tasks.
  • The Nuance of Toxicity: Even with advanced models, achieving perfect detection remains challenging. The slight differences in performance between models underscore the complexity and subjective nature of identifying toxic language.

Key Results

The paper reports the following F1 scores on the test dataset:

  • Waw-unet: 0.6251
  • RoBERTa: 0.6390
  • XLM-RoBERTa: 0.6601

While Waw-unet achieved a respectable score, the transformer-based models, particularly XLM-RoBERTa, performed better. The authors note that their best submitted result (likely from XLM-RoBERTa, given the scores) ranked 44th out of 91 submissions in the SemEval-2021 Task 5. They also mention that their models did not rely on external linguistic annotations like Part-of-Speech (PoS) tags.

Limitations and Future Work

  • Performance Gap: The Waw-unet model, while novel, did not outperform the best transformer-based models. The authors acknowledge that their best score was 4.82 points behind the first-ranked system.
  • Unsupervised Training: For future work, the authors plan to explore unsupervised training of the Waw-unet architecture on large datasets. The goal is to enable it to compete more effectively with pre-trained general language models. This suggests a direction towards leveraging larger, unlabeled text corpora to improve the model's understanding of language nuances.

Summary updated

BibTeX
@inproceedings{delil2021sefamerve,
  author = {Selman Delil and Birol Kuyumcu and Cüneyt Aksakallı},
  title = {{Sefamerve ARGE at SemEval-2021 Task 5: Toxic Spans Detection Using Segmentation Based 1-D Convolutional Neural Network Model}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure