Documenting Large Webtext Corpora: A Case Study on the Colossal Clean\n Crawled Corpus

Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, Matt Gardner

TL;DR

This paper provides the first in-depth documentation and analysis of the Colossal Clean Crawled Corpus (C4), a massive dataset used for training large language models. The authors reveal surprising data sources, identify issues like machine-generated text and benchmark contamination, and crucially, demonstrate that the dataset's filtering process disproportionately removes content related to minority groups.

What Problem Does This Paper Address?

The advent of large language models (LLMs) has led to a demand for ever-larger training datasets, often scraped from the internet. However, these massive web-scraped corpora are frequently released with minimal documentation, leaving researchers and practitioners in the dark about their origins, content, and potential biases. This lack of transparency hinders reproducible science and can lead to unintended consequences when LLMs are deployed. This paper addresses this critical gap by providing a comprehensive analysis of the C4 dataset, a prominent example of such a corpus.

Methods

The authors employ a multi-faceted approach to document and analyze the C4 dataset:

  • Provenance Analysis: They investigate the origins of the data by examining top-level domains (TLDs) and specific websites, identifying unexpected sources like patent databases and US military sites.
  • Content Analysis: They analyze the text itself to identify machine-generated content (e.g., from machine translation) and evidence of "data contamination," where examples from existing NLP benchmarks appear in the training data.
  • Filter Impact Analysis: A key contribution is evaluating the effect of the filtering process used to create C4. They specifically examine the impact of a "blocklist" filter designed to remove offensive language, analyzing which types of documents are disproportionately removed.
  • Bias Measurement: They reproduce and extend previous work on measuring demographic biases, specifically focusing on ethnicity and sentiment.

Datasets

  • Colossal Clean Crawled Corpus (C4): The primary dataset analyzed. The authors provide three versions:
  • C4.EN.NOCLEAN: The snapshot of Common Crawl identified as English, with no additional filters.
  • C4.EN.NOBLOCKLIST: C4.EN without the "bad words" blocklist filter.
  • C4.EN: The fully cleaned version of the corpus.
  • Existing NLP Benchmarks: The paper examines contamination from various NLP benchmarks, including:
  • Abstractive summarization datasets (TIFU, XSum)
    • Table-to-text generation (WikiBio)
    • Graph-to-text generation (AMR-to-text)
    • LAMA dataset (for probing knowledge completion)
    • GLUE benchmark (for general language understanding)

Key Insights and Ideas

  • Unexpected Data Sources: C4 contains a significant amount of text from sources like patent databases (e.g., patents.google.com) and US military websites, which are not typical for general language modeling.
  • Machine-Generated Content is Prevalent: The dataset includes substantial amounts of machine-generated text, such as from machine translation systems, highlighting the increasing presence of synthetic data on the web.
  • Benchmark Contamination is Real: Existing NLP benchmark examples are found in C4, sometimes at surprisingly high rates (e.g., over 50% for QNLI sentence inputs). This can lead to inflated performance metrics and an inaccurate understanding of model capabilities.
  • Blocklist Filtering is Problematic: The "bad words" blocklist, intended to remove offensive content, disproportionately filters out documents related to minority identities, including sexual orientations and African American English. This is a form of "allocational harm."
  • Bias is Embedded and Amplified: The paper provides evidence that C4 exhibits sentiment bias, with "Jewish" being associated with more positive sentiment than "Arab." This bias likely stems from the web data and can be amplified by LLMs trained on it.

Key Results

  • Domain Distribution: The top represented TLDs in C4.EN are .com, .org, and .net. Notably, .gov and .mil domains are also significant sources.
  • Website Representation: patents.google.com is the most represented website, followed by Wikipedia and news outlets.
  • Benchmark Overlap: Target texts from datasets like XSum and TIFU-short appear in C4.EN at rates of 15.49% and 24.88%, respectively. For GLUE, QNLI sentence inputs show over 53% overlap.
  • Filter Exclusion: African American English (AAE) and Hispanic-aligned English are removed at significantly higher rates (42% and 32%) by the blocklist filter compared to White-aligned English (6.2%).
  • Sentiment Bias: The study found a positivity gap of 7.5% between "Jewish" and "Arab" in C4.EN, with "Jewish" having a higher proportion of positive sentiment.

Limitations and Future Work

The authors acknowledge several limitations:

  • Internet Archive Proxy: Using the Internet Archive's indexing date as a proxy for utterance date is not perfect.
  • Geolocation Imperfections: IP-based geolocation has inherent limitations.
  • Causality of Bias: While they show correlation, they haven't definitively proven a causal link between C4's biases and downstream model behavior.
  • Completeness of Analysis: They recognize that only a subset of potential issues (like personally identifiable information or copyrighted text) were examined.
  • Language Generalization: The analyses are primarily focused on English and may not generalize to other languages.

Future work could involve developing more nuanced filtering mechanisms beyond simple blocklists, creating more transparent and auditable datasets, and further investigating the causal links between dataset biases and model behavior. The authors also recommend providing a mechanism for reporting issues found in the dataset.

Summary updated

Abstract

Large language models have led to remarkable progress on many NLP tasks, and\nresearchers are turning to ever-larger text corpora to train them. Some of the\nlargest corpora available are made by scraping significant portions of the\ninternet, and are frequently introduced with only minimal documentation. In\nthis work we provide some of the first documentation for the Colossal Clean\nCrawled Corpus (C4; Raffel et al., 2020), a dataset created by applying a set\nof filters to a single snapshot of Common Crawl. We begin by investigating\nwhere the data came from, and find a significant amount of text from unexpected\nsources like patents and US military websites. Then we explore the content of\nthe text itself, and find machine-generated text (e.g., from machine\ntranslation systems) and evaluation examples from other benchmark NLP datasets.\nTo understand the impact of the filters applied to create this dataset, we\nevaluate the text that was removed, and show that blocklist filtering\ndisproportionately removes text from and about minority individuals. Finally,\nwe conclude with some recommendations for how to created and document web-scale\ndatasets from a scrape of the internet.\n
BibTeX
@inproceedings{dodge2021documenting,
  author = {Jesse Dodge and Maarten Sap and Ana Marasović and William Agnew and Gabriel Ilharco and Dirk Groeneveld and Margaret Mitchell and Matt Gardner},
  title = {{Documenting Large Webtext Corpora: A Case Study on the Colossal Clean\textbackslash\{\}n Crawled Corpus}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.98},
}
Figure