Toxicity Classification in Ukrainian

Daryna Dementieva, Valeriia Khylenko, Nikolay Babakov, Georg GrohNAACL 2024

TL;DR

This paper tackles the critical challenge of toxic language detection in Ukrainian, a language underserved by existing resources. The authors introduce the first Ukrainian toxicity classification corpus, developed through a combination of translation, keyword filtering, and crowdsourcing. They also evaluate various cross-lingual transfer learning methods, offering valuable insights into building robust toxicity classifiers for low-resource languages.

What Problem Does This Paper Address?

The proliferation of online content has made toxic language detection a crucial task for ensuring safe and fair digital environments. While significant progress has been made for languages like English, many other languages, including Ukrainian, lack sufficient labeled data and robust classification models. This resource scarcity hinders the development of effective tools to combat online abuse and misinformation in these languages. The paper aims to bridge this gap by creating a Ukrainian toxicity dataset and exploring efficient methods for building classifiers, particularly focusing on cross-lingual transfer learning techniques.

Methods

The researchers employed a multi-pronged approach to both create a dataset and evaluate classification methods:

  • Data Acquisition:
    • Translation: An existing English toxicity corpus (Jigsaw dataset) was translated into Ukrainian.
  • Semi-synthetic Data: Ukrainian tweets were filtered using a predefined list of toxic keywords, supplemented with news and fiction texts for non-toxic examples.
  • Crowdsourcing: A dedicated annotation effort was undertaken using the Toloka platform, where Ukrainian speakers labeled tweets for toxicity. This involved rigorous quality control measures for annotators.
  • Cross-lingual Knowledge Transfer:
    • Backtranslation: An English toxicity classifier was used by translating Ukrainian text into English.
  • LLM Prompting: Large Language Models (LLMs) like LLaMa-2 and Mistral were prompted to classify Ukrainian text directly, without fine-tuning.
  • Adapter Training: Language-specific adapter layers were trained on top of multilingual models (like XLM-R) for the Ukrainian task.
  • Supervised Learning:
    • Fine-tuning: Models were fine-tuned on the acquired Ukrainian datasets (translated, semi-synthetic, and crowdsourced).

Datasets

The paper introduces and utilizes the following datasets:

  • English Jigsaw Dataset: Used as the source for translation into Ukrainian.
  • Ukrainian Tweets Corpus (Bobrovnyk, 2019a): The primary source for semi-synthetic data creation and crowdsourced annotation.
  • News and Fiction UD Ukrainian IU Dataset (Kotsyba et al., 2016): Used to supplement non-toxic examples in the semi-synthetic dataset.
  • Newly Introduced Ukrainian Toxicity Datasets: The paper presents three distinct datasets created through the methods described above:
  • Translated Dataset: Ukrainian text derived from English toxic comments.
  • Semi-synthetic Dataset: Ukrainian text filtered by toxic keywords.
  • Crowdsourced Dataset: Manually annotated Ukrainian text, considered the most representative of real-world toxic language.

Key Insights and Ideas

  • Crowdsourcing is King (for accuracy): The study strongly suggests that manually annotated data (crowdsourced) yields the highest performance for toxicity classification, as it best captures the nuances of real-world toxic language.
  • Translation's Double-Edged Sword: While translating an English dataset is a viable and resource-efficient baseline, the translated data may lose some toxicity, leading to lower performance compared to natively annotated data. However, fine-tuning on this translated data can still yield robust results.
  • LLMs Show Promise, but Need Careful Prompting: LLM prompting (zero-shot/few-shot) offers a fine-tuning-free approach, but performance can vary significantly depending on the LLM and the specific test set. Mistral showed better results than LLaMa-2 on some sets, but Backtranslation outperformed them on the crucial crowdsourced data.
  • Adapter Training Offers Efficiency: Adapter training provides a parameter-efficient way to adapt multilingual models, showing competitive results without the need to fine-tune the entire model.
  • Backtranslation as a Strong Unsupervised Baseline: The backtranslation method, despite requiring multiple inference steps, proved to be a surprisingly strong unsupervised baseline, outperforming LLM prompting on the most representative test sets.

Key Results

  • Highest Performance: The model fine-tuned on the crowdsourced dataset achieved the highest performance, demonstrating the value of human annotation.
  • Robustness of Translated Data: Fine-tuning on the translated dataset still provided the best results on the annotated test set, indicating its utility as a baseline.
  • LLM Prompting Variability: Mistral outperformed LLaMa-2 on the semi-synthetic test set but performed poorly on the translated and crowdsourced sets.
  • Backtranslation's Strength: Backtranslation achieved top results on the translated and crowdsourced datasets, highlighting its effectiveness for real Ukrainian toxic data.
  • XLM-R Fine-tuning: When fine-tuned on the crowdsourced data, XLM-R exhibited near-perfect performance.

Limitations and Future Work

  • Definition of Toxicity: The study defines toxicity primarily based on obscene lexicon and vulgar/profane language, acknowledging that it does not cover other forms of harmful language like hate speech, sarcasm, racism, or sexism.
  • Resource-Rich Language Focus: The research primarily used English as the resource-rich language. Future work could explore other linguistically closer languages (e.g., Polish, Croatian) for cross-lingual transfer.
  • Model Availability: The authors have made their obtained data and models publicly available, encouraging further research and development in this area.

The paper concludes by emphasizing the importance of sharing these resources to foster the development of fairer and safer multilingual LLMs.

Summary updated

BibTeX
@inproceedings{dementieva2024toxicity,
  author = {Daryna Dementieva and Valeriia Khylenko and Nikolay Babakov and Georg Groh},
  title = {{Toxicity Classification in Ukrainian}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure