MultiParaDetox: Extending Text Detoxification with Parallel Data to New Languages

Daryna Dementieva, Nikolay Babakov, Alexander PanchenkoNAACL 2024 · DOI

TL;DR

This paper introduces MultiParaDetox, a novel pipeline for automatically collecting parallel text detoxification datasets in new languages. They demonstrate its effectiveness by creating datasets for Russian, Ukrainian, and Spanish, and show that fine-tuning models on these parallel corpora significantly outperforms existing methods, paving the way for more equitable and safer AI across languages.

What Problem Does This Paper Address?

Text detoxification aims to transform toxic or offensive language into a neutral, polite style while preserving the original meaning. While effective methods have emerged, they heavily rely on parallel datasets (toxic text paired with its detoxified version). However, existing parallel datasets are primarily limited to English. This creates a significant gap, hindering the development of robust text detoxification models for the vast majority of the world's languages, leading to potential biases and safety concerns in AI systems deployed globally. The authors tackle this by proposing a scalable, multilingual approach to parallel dataset collection.

Methods

The core contribution is the MultiParaDetox pipeline, an adaptation of the existing ParaDetox collection method. This pipeline involves three key tasks for crowdsourced annotation:

  1. Rewrite text in a polite way: Annotators rephrase toxic text into a neutral style, preserving meaning, or skip if impossible.
  2. Do these sentences mean the same? Annotators verify if the original toxic text and its detoxified version convey the same core meaning.
  3. Is this text offensive? Annotators confirm that the detoxified text is indeed non-offensive.

The pipeline is designed for language adaptation, allowing it to be applied to new languages by leveraging automated translation for initial setup and then refining with native speaker proofreading. They also detail strategies for toxic corpus preparation, including keyword searches and filtering existing toxicity datasets.

Datasets

The paper introduces new parallel text detoxification datasets for:

  • Russian
  • Ukrainian
  • Spanish

These datasets were collected using the MultiParaDetox pipeline via the Toloka crowdsourcing platform. The authors provide statistics on the size and characteristics of these new datasets, noting that they collected hundreds of pairs per language. They also mention using existing toxicity datasets for Russian and Spanish, and filtering Ukrainian tweets for explicit toxicity.

Key Insights and Ideas

  • Parallel data is crucial for state-of-the-art text detoxification: The study strongly reaffirms that fine-tuning models on parallel corpora yields significantly better results than unsupervised methods or models trained on non-parallel data.
  • Crowdsourcing is effective for multilingual dataset creation: The MultiParaDetox pipeline demonstrates that crowdsourcing can be successfully adapted to collect high-quality parallel detoxification data across diverse languages, even with cultural nuances.
  • LLMs show promise but require careful prompting and fine-tuning: While large language models (LLMs) like LLaMa can achieve high Style Transfer Accuracy (STA), they often struggle with content similarity (SIM) and can sometimes hallucinate or produce text in the wrong language. Fine-tuning on parallel data remains superior.
  • Cross-lingual transfer is challenging but possible: Models fine-tuned on translated English data performed poorly, highlighting the difficulty of direct translation for complex linguistic and cultural aspects of toxicity. This underscores the need for native-language parallel data.

Key Results

  • Models fine-tuned on the newly created MultiParaDetox datasets (Russian, Ukrainian, Spanish) outperformed unsupervised baselines and LLMs by a significant margin across all evaluation metrics (Style Transfer Accuracy, Content Similarity, Fluency, and a combined Joint metric).
  • For Russian, the fine-tuned model achieved a Joint score of 0.492, significantly higher than other methods.
  • For Ukrainian, the fine-tuned model achieved a Joint score of 0.511, also leading the pack.
  • For Spanish, the fine-tuned model achieved a Joint score of 0.239, demonstrating effectiveness despite a smaller dataset size.
  • LLaMa, when prompted, showed high STA but low SIM scores, indicating it could make text polite but often changed the meaning.
  • Models fine-tuned on translated English data performed poorly, with Joint scores as low as 0.000 for Ukrainian.

Limitations and Future Work

  • Focus on Explicit Toxicity: The current work and datasets primarily address explicit forms of toxicity (e.g., profanity, slurs). Implicit toxicity (e.g., sarcasm, subtle bias) is acknowledged as a more challenging area requiring different definitions and data collection strategies.
  • Uneven Data Distribution: The collected datasets exhibited uneven sample ratios due to the sequential nature of the experiments and available resources. The authors plan to share task instructions to encourage community contributions for further data collection.
  • Minimal Data for Fine-tuning: Future work could explore the minimum amount of parallel data required to train a robust text detoxification model.
  • Multilingual Model Exploration: While they presented experiments for a single multilingual model, further research is needed to optimize cross-lingual knowledge transfer and explore transfer between languages from neighboring families.

Summary updated

BibTeX
@inproceedings{dementieva2024multiparadetox,
  author = {Daryna Dementieva and Nikolay Babakov and Alexander Panchenko},
  title = {{MultiParaDetox: Extending Text Detoxification with Parallel Data to New Languages}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-short.12},
}
Figure