Toxicity of the Commons: Curating Open-Source Pre-Training Data

Catherine Arnett, Eliot Jones, Ivan P. Yamshchikov, Pierre-Carl LanglaisarXiv 2024 · DOI · arXiv

Abstract

As the gap between open-weight and closed-source large language models (LLMs) narrows, the focus shifts to the "final frontier" of openness: the training data itself. This paper addresses the ethical and technical challenges of cleaning public domain data—such as historical books and newspapers—without compromising the diversity of the "commons." We introduce ToxicCommons, a 2-million-sample annotated dataset, and Celadon, a specialized classifier that outperforms standard filters on historical and OCR-noisy text by prioritizing "detoxification" over simple deletion.

Introduction

Proprietary datasets remain a significant barrier to truly open AI development. While models like Llama 3.1 provide high capabilities, the data used to train them is often hidden. Public domain data offers a path forward, but existing toxicity filters, such as Google's Perspective API, are often "black boxes" trained on modern web text. These filters struggle with historical context and OCR noise, frequently flagging non-toxic literary descriptions of violence or historical events as harmful. This "context blindness" leads to the unnecessary deletion of valuable data, reducing representation and linguistic diversity.

Methodology

The researchers developed a three-stage pipeline to address these issues:

  1. LLM-in-the-Loop Annotation: Llama 3.1 was used to score text across five dimensions—race, gender/sexuality, religion, ability, and violence—on a 0–3 severity scale, grounded by human-verified samples.
  2. The Celadon Classifier: To enable scalability, the LLM’s knowledge was distilled into Celadon, a 5-headed classifier based on DeBERTa-v3-small. A custom weighted loss function was employed to handle the inherent class imbalance of toxic samples.
  3. Balanced Mitigation: Instead of a binary "keep or delete" approach, data is categorized into three tiers:
  4. No Toxicity: Retained as-is.
  5. Mild Toxicity: Retained with a content warning.
  6. Toxic Content: Synthetically re-written ("detoxified") by an LLM to remove harmful bias while preserving original style and tone.

Data Distribution and Insights

Analysis of the Common Corpus—which includes 13 cultural heritage datasets across 9 languages—reveals that "Violence and Abuse" is the most frequently flagged category.

Proportion of toxicity scores across different bias categories in US Public Domain Newspapers. Violence is significantly more prevalent than other categories.

As shown in Figure 1, violence scores are notably higher than other bias categories in historical newspaper data. However, the authors note that filters are often overly sensitive to neutral historical records of conflict. By using a "detox over delete" philosophy, the researchers preserve the linguistic richness of the dataset while reducing the "prior" for toxic model behavior.

Performance Evaluation

The Celadon classifier demonstrates significant efficiency gains, operating over 40x faster than LLM-based annotation. On an NVIDIA H100, it processed 100,000 samples in just 5 minutes.

Performance metrics for the Celadon classifier, showing high precision and weighted accuracy.

As detailed in the performance results, Celadon achieves high F1 scores and a Weighted Accuracy (WA) of approximately 0.73. This indicates the model successfully learns to distinguish between different levels of severity rather than simply predicting the majority "not toxic" class.

Conclusion

The Toxicity of the Commons project provides a blueprint for ethical data curation in the open-source community. By moving beyond simple deletion toward context-aware detoxification, researchers can build robust datasets that respect historical context. Future work will focus on expanding linguistic diversity beyond Western European languages and refining synthetic re-writing processes to ensure factual accuracy.

Summary updated

Abstract

Open-source large language models are becoming increasingly available and popular among researchers and practitioners. While significant progress has been made on open-weight models, open training data is a practice yet to be adopted by the leading open-weight models creators. At the same time, there researchers are working to make language models safer. We propose a data curation pipeline to reduce harmful outputs by models trained on public domain data. There are unique challenges to working with public domain data, as these sources differ from web text in both form and content. Many sources are historical documents and are the result of Optical Character Recognition (OCR). Consequently, current state-of-the-art approaches to toxicity filtering are often infeasible or inappropriate for open data models. In this paper, we introduce a new fully open-source pipeline for open-data toxicity filtering. Our contributions are threefold. We create a custom training dataset, ToxicCommons, which is composed of texts which have been classified across five different dimensions (racial/origin-based, gender/sex-based, religious, ability-based discrimination, and violence). We use this dataset to train a custom classifier, Celadon, that can be used to detect toxic content in open data more efficiently at a larger scale. Finally, we describe the balanced approach to content filtration that optimizes safety filtering with respect to the filtered data available for training.
BibTeX
@misc{arnett2024toxicity,
  author = {Catherine Arnett and Eliot Jones and Ivan P. Yamshchikov and Pierre-Carl Langlais},
  title = {{Toxicity of the Commons: Curating Open-Source Pre-Training Data}},
  year = {2024},
  eprint = {2410.22587},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.22587},
}
Figure