Abstract
As the gap between open-weight and closed-source large language models (LLMs) narrows, the focus shifts to the "final frontier" of openness: the training data itself. This paper addresses the ethical and technical challenges of cleaning public domain data—such as historical books and newspapers—without compromising the diversity of the "commons." We introduce ToxicCommons, a 2-million-sample annotated dataset, and Celadon, a specialized classifier that outperforms standard filters on historical and OCR-noisy text by prioritizing "detoxification" over simple deletion.
Introduction
Proprietary datasets remain a significant barrier to truly open AI development. While models like Llama 3.1 provide high capabilities, the data used to train them is often hidden. Public domain data offers a path forward, but existing toxicity filters, such as Google's Perspective API, are often "black boxes" trained on modern web text. These filters struggle with historical context and OCR noise, frequently flagging non-toxic literary descriptions of violence or historical events as harmful. This "context blindness" leads to the unnecessary deletion of valuable data, reducing representation and linguistic diversity.
Methodology
The researchers developed a three-stage pipeline to address these issues:
- LLM-in-the-Loop Annotation: Llama 3.1 was used to score text across five dimensions—race, gender/sexuality, religion, ability, and violence—on a 0–3 severity scale, grounded by human-verified samples.
- The Celadon Classifier: To enable scalability, the LLM’s knowledge was distilled into Celadon, a 5-headed classifier based on DeBERTa-v3-small. A custom weighted loss function was employed to handle the inherent class imbalance of toxic samples.
- Balanced Mitigation: Instead of a binary "keep or delete" approach, data is categorized into three tiers:
- No Toxicity: Retained as-is.
- Mild Toxicity: Retained with a content warning.
- Toxic Content: Synthetically re-written ("detoxified") by an LLM to remove harmful bias while preserving original style and tone.
Data Distribution and Insights
Analysis of the Common Corpus—which includes 13 cultural heritage datasets across 9 languages—reveals that "Violence and Abuse" is the most frequently flagged category.
As shown in Figure 1, violence scores are notably higher than other bias categories in historical newspaper data. However, the authors note that filters are often overly sensitive to neutral historical records of conflict. By using a "detox over delete" philosophy, the researchers preserve the linguistic richness of the dataset while reducing the "prior" for toxic model behavior.
Performance Evaluation
The Celadon classifier demonstrates significant efficiency gains, operating over 40x faster than LLM-based annotation. On an NVIDIA H100, it processed 100,000 samples in just 5 minutes.
As detailed in the performance results, Celadon achieves high F1 scores and a Weighted Accuracy (WA) of approximately 0.73. This indicates the model successfully learns to distinguish between different levels of severity rather than simply predicting the majority "not toxic" class.
Conclusion
The Toxicity of the Commons project provides a blueprint for ethical data curation in the open-source community. By moving beyond simple deletion toward context-aware detoxification, researchers can build robust datasets that respect historical context. Future work will focus on expanding linguistic diversity beyond Western European languages and refining synthetic re-writing processes to ensure factual accuracy.
Summary updated