Identifying Rare Languages in Common Crawl Data is a Needles-in-a-Haystack Problem

Rasul Dent, Pedro Ortiz Suárez, Thibault Clérice, Benoît Sagot2025 · DOI

TL;DR

This paper tackles the challenge of finding rare languages within massive web crawls like Common Crawl, which are typically dominated by high-resource languages. They introduce a "Needles-in-a-Haystack" approach using a two-pass filtering system that efficiently identifies and extracts data for underrepresented languages, significantly outperforming existing methods in speed and recall for French-based Creoles.

What Problem Does This Paper Address?

The internet is a treasure trove of linguistic data, but it's heavily skewed towards a few dominant languages. Existing tools for language identification (LID) and corpus creation, like those used for OSCAR or Fineweb, are often designed for multi-class classification across many languages. While effective for common languages, they struggle with "long-tail" languages – those with very little online presence. This is particularly true for language varieties that share significant lexical overlap with more common languages, such as French-based Creoles, which can be easily misclassified. The authors frame this as a "Needles-in-a-Haystack" problem: finding small amounts of valuable data amidst a vast ocean of irrelevant content. They aim to develop a more efficient and scalable method to create corpora for these rare languages, which are crucial for linguistic research and NLP development.

Methods

The core of their approach is a two-pass filtering strategy:

  • Document-level Filtering: This initial pass uses a "Bag-of-Types" strategy. It leverages whitelists of distinctive words or character patterns for target languages and blacklists to quickly discard irrelevant documents. This is designed to be very fast, aiming to eliminate the vast majority of "hay" (unwanted documents) with minimal computational cost. They also explore scoring mechanisms to rank documents based on their likelihood of containing target language content.
  • Line-level Filtering: After the initial document-level pass, a second, more refined pass operates at the line level. This helps to isolate specific sentences or phrases within documents, further cleaning the data and improving precision. They adapt wordlist-based approaches for this stage, normalizing scores by line length to better handle short but relevant lines.

The overall pipeline is designed for efficiency, prioritizing speed and recall in the initial stages to handle massive datasets.

Datasets

  • Common Crawl: The primary source of raw data for their experiments. They specifically use the December 2024 snapshot (CC-MAIN-2024-51), a 21 TB dataset.
  • Wikipedia: Used to create a clean benchmark dataset for evaluating speed, recall, and false positive rates. They selected 10,000 entries, with a significant imbalance (9800 French, 200 French Guianese Creole) to mimic real-world class imbalances.
  • Existing Filtered Corpora: They evaluate their method's recall on "clean" and "noisy" portions of:
  • MADLAD-400
    • GlotCC
    • FineWeb-2

Key Insights and Ideas

  • Reframing LID as Data Mining: The paper argues that for rare languages, LID should be treated as a data mining problem rather than a standard multi-class classification task. This shift in perspective is key to developing efficient filtering strategies.
  • The Power of Whitelists/Blacklists for Speed: Simple, lexicon-based whitelists and blacklists are surprisingly effective and extremely fast for initial document-level filtering, especially for languages with distinct orthographies like French-based Creoles.
  • Class Imbalance is a Feature, Not a Bug: The massive imbalance between high-resource and low-resource languages in Common Crawl is precisely what makes the "Needles-in-a-Haystack" approach viable. The vast majority of data can be discarded quickly.
  • Efficiency is Paramount for Rare Languages: Traditional LID models, while accurate, are too slow for processing the sheer volume of data required to find sufficient examples of rare languages. Their method prioritizes speed to make this feasible.
  • Creole Languages as a Test Case: French-based Creoles, with their shared lexicon with French but distinct morphosyntax, present a unique challenge that their method is well-suited to address.

Key Results

  • Speed: Their document-level filtering pipeline processed 21 TB of raw Common Crawl data in 2-4.5 hours on a medium-sized cluster, significantly faster than existing methods like GlotCC (estimated 340 hours for a similar snapshot).
  • Recall: Achieved high recall (e.g., 79.0% for French Guianese Creole with a single-language setting) on clean Wikipedia data, and demonstrated strong performance on noisy real-world corpora.
  • Efficiency: The method is described as 50-100x faster than GlotCC's reported wall time for pre-filtering.
  • Data Discovery: Their filtering significantly increased the amount of readily usable web data for several Creole languages, sometimes doubling corpora sizes by recovering data from "removed" (noisy) portions of existing datasets.
  • Thresholding Impact: They showed that adjusting the whitelist score threshold allows for a trade-off between recall and precision, effectively eliminating most French documents while retaining a high percentage of Creole content.

Limitations and Future Work

  • Harmful Content: The paper acknowledges the risk of encountering harmful or derogatory content and notes that their current blacklists are basic (e.g., filtering machine-translated pornography). They suggest the need for more sophisticated, language-specific tools for identifying subtle forms of harmful content.
  • Generalizability to Other Language Types: While effective for French-based Creoles, the whitespace-tokenization approach might be less effective for languages with different orthographic or morphological structures. They mention the need for more robust methods for languages with less distinct word boundaries or those prone to "ANTSPEAK" (excessive whitespace).
  • Fine-grained Content Filtering: Implementing more advanced content filtering for nuanced issues (like generalizations about groups) is a future direction.
  • Exploration of Other Languages: The current work focuses on French-based Creoles, but the approach is intended to be generalizable to other rare languages.
  • Internet Bandwidth as a Bottleneck: For extremely efficient LID, data transfer and decompression times can become the limiting factor, suggesting in-situ processing might be necessary.

Summary updated

Abstract

International audience
BibTeX
@misc{dent2025identifying,
  author = {Rasul Dent and Pedro Ortiz Suárez and Thibault Clérice and Benoît Sagot},
  title = {{Identifying Rare Languages in Common Crawl Data is a Needles-in-a-Haystack Problem}},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.77},
}
Figure