Towards Safer Pretraining: Analyzing and Filtering Harmful Content in Webscale Datasets for Responsible LLMs

Sai Krishna Mendu, Harish Yenala, Aditi Gulati, Shanu Kumar, Parag AgrawalIJCAI 2025 · DOI

TL;DR

This paper tackles the critical issue of harmful content in the massive datasets used to pretrain Large Language Models (LLMs). The authors introduce a novel, three-dimensional taxonomy for classifying harmful content and develop tools like a prompt-based classifier (TTP) and a transformer model (HarmFormer) to filter this content effectively, paving the way for more responsible AI development.

What Problem Does This Paper Address?

Large Language Models (LLMs) are trained on vast amounts of text data scraped from the internet. While this data is crucial for their impressive capabilities, it also contains a significant amount of harmful content, including hate speech, misinformation, and biased narratives. Training LLMs on such unfiltered data risks perpetuating these negative aspects, leading to toxic outputs and undermining trust in AI applications. Existing content moderation tools often struggle with the scale and complexity of web-scale data, using simplistic binary classifications and failing to handle long-form text or nuanced intent. This paper aims to bridge this gap by providing a more sophisticated approach to identifying and filtering harmful content during the pretraining phase.

Methods

The core of the paper's methodology lies in its novel three-dimensional taxonomy for classifying content. This taxonomy moves beyond simple "safe" vs. "toxic" labels by introducing:

  • Safe: Non-harmful material.
  • Topical: Contextual discussions that are important for LLM capabilities but might contain keywords associated with harm (e.g., discussions about mental health resources).
  • Toxic: Explicitly harmful content.

This tripartite structure is applied across five harm categories: Hate & Violence, Ideological Harm, Sexual Content, Illegal Activities, and Self-Inflicted Harm.

To operationalize this taxonomy, the authors developed:

  • TTP (Topical and Toxic Prompt): A prompt-based classifier, fine-tuned using GPT-4 Omni, designed to distinguish between discussing harmful topics and actively promoting them.
  • HarmFormer: A LongFormer-based transformer model specifically optimized for long-form content moderation, achieving high accuracy while preserving topical text.

Datasets

The paper utilizes several large-scale datasets for analysis and model training:

  • Common Crawl, C4, and FineWeb: These are the primary web-scale datasets analyzed for the prevalence of harmful content.
  • TTP-Eval Set: A newly introduced benchmark of 491 annotated web pages, designed to evaluate prompt-based classifiers for long-text moderation. It spans diverse harms and document lengths.
  • HAVOC (Harmful Abstractions and Violations in Open Completions) Benchmark: Another novel benchmark created to evaluate LLM safety by measuring toxic generation tendencies in open-ended responses to adversarial prompts. It consists of 10,376 snippets from web pages.
  • OpenAI Moderation Dataset: Used for comparing the performance of their proposed methods against existing tools.

Key Insights and Ideas

  • The "Topical" dimension is crucial: Simply filtering out content with keywords related to harm can inadvertently remove valuable, contextually important information that LLMs need to be capable. The paper highlights that many discussions about sensitive topics (like mental health or societal bias) are topical, not toxic.
  • Existing moderation tools are insufficient for web-scale data: Their limitations in handling long-form text and nuanced intent mean that harmful content can easily slip through.
  • Harmful content is prevalent, but not overwhelming: While toxic content constitutes a significant portion (2.1-4.1%) of web corpora, topical content is also substantial (3.4-10.6%), underscoring the need for careful filtering.
  • Sexual harm is a particularly problematic area: The analysis shows it's the most leaking harm category when LLMs are provoked, with Common Crawl being heavily dominated by explicit pornography.
  • Adversarial prompts reveal LLM vulnerabilities: The HAVOC benchmark demonstrates that even seemingly benign prompts can elicit harmful responses from LLMs, especially when the prompt itself contains toxic elements.

Key Results

  • HarmFormer achieves high accuracy: The HarmFormer model demonstrates strong performance, achieving an 85% F1-score on the toxic dimension of their test set.
  • TTP outperforms existing tools: On the OpenAI Moderation Dataset, TTP achieved a Precision of 0.72 and Recall of 0.91, outperforming Perspective API and Llama Guard in various settings.
  • Significant toxic content in web-scale datasets: Analysis of 3 million web pages revealed that Common Crawl has the highest proportion of toxic content (4.1%), driven by sexual content. C4 and FineWeb, which employ some filtering, show lower toxicity but still notable levels.
  • LLMs leak toxic content: The HAVOC benchmark revealed an overall leakage rate of 26.7% across models, with provocative inputs leading to a 76% leakage rate. Sexual harm was identified as the most problematic category.

Limitations and Future Work

The authors acknowledge several limitations:

  • English-centric: The current study focuses solely on English data, leaving multilingual analysis for future work.
  • Culture-specific sensitivities: The proposed taxonomy, while broad, may not fully account for culture-specific nuances where certain actions might be considered harmful in one context but not another.
  • Dataset purity impact: They plan to investigate the impact of dataset purity on model behavior and performance by training models on dimensionally filtered datasets and comparing them to models trained on unclean datasets. This will help understand the interplay between data cleanliness and model behavior when encountering toxic inputs across multiple harms.

Summary updated

BibTeX
@inproceedings{mendu2025towards,
  author = {Sai Krishna Mendu and Harish Yenala and Aditi Gulati and Shanu Kumar and Parag Agrawal},
  title = {{Towards Safer Pretraining: Analyzing and Filtering Harmful Content in Webscale Datasets for Responsible LLMs}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/53},
}
Figure