Validating and Exploring Large Geographic Corpora

Jonathan DunnCOLING 2024

TL;DR

This paper tackles the challenge of creating high-quality, representative web corpora for diverse languages and populations by systematically evaluating data cleaning methods. They found that while cleaning improves corpus validity, its impact is uneven across languages, potentially excluding under-represented groups.

What Problem Does This Paper Address?

Large-scale web corpora are invaluable for Natural Language Processing (NLP), but their creation from sources like the Common Crawl involves significant cleaning steps. These steps, while aiming to remove noise and improve data quality, can inadvertently filter out linguistic diversity, particularly for less common languages and populations. Existing methods often lack a systematic evaluation of how each cleaning stage impacts different linguistic groups. This paper addresses this gap by investigating the effects of three specific cleaning methods on a large geographic web corpus, aiming to understand their impact on both overall corpus quality and the representation of diverse language-country pairs.

Methods

The authors start with a massive 427 billion word web corpus (CGLU 4.3) derived from the Common Crawl. They then apply a sequence of three increasingly sophisticated cleaning methods:

  1. Language Identification Agreement (CGLU 5.0): Instead of relying on a single language identification model, they use multiple models to triangulate language labels. Documents where models disagree are removed, ensuring higher confidence in the assigned language.
  2. Hash-based Deduplication (CGLU 5.1): This method removes duplicate or near-duplicate text segments. They employ a hash-based approach at the sentence level, which is more aggressive than previous paragraph-level methods, aiming to eliminate boilerplate content and highly similar documents.
  3. Location-Specific Outlier Detection (CGLU 5.2): This technique identifies and removes noisy samples that are statistically unusual for a specific language-country pair. Unlike general outlier detection, this method trains language models on sub-corpora specific to each language-country pair (e.g., Swiss German), making it sensitive to regional linguistic variations.

The impact of each step is evaluated by comparing the resulting corpora to baseline datasets using corpus similarity measures.

Datasets

  • Primary Dataset: Corpus of Global Language Use (CGLU 4.3), a 427 billion word web corpus derived from the Common Crawl.
  • Intermediate Corpora: CGLU 5.0, CGLU 5.1, and CGLU 5.2, representing the corpus after each cleaning stage.
  • Benchmark Corpora for Validation:
    • Language Identification Corpora: Seven independent sources of text known to be in specific languages (e.g., Bible translations, JW 300 dataset, Open Subtitles). These are used to validate language labels.
  • Geo-located Tweets (TWGLU): A 23 billion word corpus of tweets collected from specific countries. This is used to validate the geographic representation of the corpora.

Key Insights and Ideas

  • Uneven Impact of Cleaning: The most striking finding is that cleaning methods do not affect all languages and populations equally. While cleaning generally improves corpus validity, it can disproportionately reduce the size of corpora for certain languages.
  • Deduplication as a Major Driver of Change: The hash-based deduplication step had the most significant impact on the corpus distribution, leading to substantial reductions in languages like Chinese and Japanese. This suggests that current deduplication methods might be too aggressive for certain linguistic contexts.
  • Outlier Detection's Nuance: Location-specific outlier detection, while having the smallest impact on corpus size, proved effective at removing noisy samples and improving corpus similarity to benchmarks. This method's strength lies in its ability to account for regional linguistic variations.
  • Risk of Excluding Under-represented Groups: The paper highlights a critical trade-off: cleaning methods designed to remove noise can also inadvertently remove valuable linguistic diversity from under-represented populations, potentially exacerbating existing inequalities in NLP.

Key Results

  • The initial corpus (CGLU 4.3) of 427 billion words was reduced to 332 billion (CGLU 5.0) after language identification agreement, 219 billion (CGLU 5.1) after deduplication, and 213 billion (CGLU 5.2) after outlier detection.
  • Language identification agreement removed 95 billion words (22.2% of the original data).
  • Hash-based deduplication removed an additional 113 billion words (34% of the data after language validation).
  • Outlier detection removed 6.35 billion words (approximately 2.9% of the data).
  • Corpus similarity measures showed that each cleaning step generally increased the similarity of the geographic corpora to the benchmark datasets, indicating improved validity.
  • The Pearson correlation between language-country sub-corpora sizes after language validation was 0.99, but dropped to 0.93 after deduplication, indicating a significant shift in distribution. This correlation returned to 0.99 after outlier detection.

Limitations and Future Work

  • Uneven Representation: The authors acknowledge that even with these cleaning steps, geographic corpora might not perfectly represent all sub-populations within a given location, as certain demographic segments are more likely to contribute to web data.
  • Deduplication Equity: The paper calls for future work to investigate more equitable cross-linguistic deduplication methods, as the current hash-based approach has an implicit side-effect of reducing some languages more than others.
  • Language Identification Challenges: The paper notes that language identification remains a challenge for less-common languages, non-standard varieties, and informal registers, which can impact corpus quality.
  • Benchmark Limitations: While using geo-referenced tweets as a benchmark is valuable, it's acknowledged that they represent a specific register and might not fully capture all linguistic variations.

The authors suggest that their work provides a systematic evaluation of corpus creation methods and introduces a cleaner, more usable multi-lingual geographic corpus, paving the way for more representative NLP resources.

Summary updated

BibTeX
@inproceedings{dunn2024validating,
  author = {Jonathan Dunn},
  title = {{Validating and Exploring Large Geographic Corpora}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure