Script Pollution and Contamination in Multilingual Corpora

64 concepts49 papers2 proposals

Modern natural language processing relies heavily on massive, web-scale corpora to train multilingual language models. On paper, these datasets boast coverage of hundreds of languages, seemingly bridging the digital divide for low-resource and minority languages. However, manual audits of these datasets reveal a starkly different reality: the data representing the long tail of human languages is often severely contaminated with noise, gibberish, and dominant global languages.

This discrepancy is driven by systemic failures in automated data collection pipelines. Understanding why these pipelines fail requires looking closely at the mechanisms of cross-lingual data pollution, script mismatch, and the limitations of automated language identification.

The Anatomy of Script Pollution

One of the most pervasive forms of data contamination is script pollution. This occurs when a dataset pipeline categorizes text as belonging to a specific low-resource language, but the actual text is written in an entirely different script—most frequently the Latin script of English or other dominant European languages.

As detailed by Constantine Lignos et al. in Toward More Meaningful Resources for Lower-resourced Languages, automatically generated "silver-standard" datasets frequently suffer from this issue. When automated harvesters extract entity labels or text strings for languages that natively use non-Latin scripts (such as Arabic, Devanagari, or Cyrillic), they often encounter missing data fields. Instead of leaving these fields blank, automated alignment tools default to pulling the English or Latin-script fallback text.

This creates what the authors term a deceptive appearance of data density, or representation washing: a dataset appears to contain thousands of tokens for a rare language, but the content is functionally English. A model trained on this data encounters Latin characters where it should see native scripts, rendering it incapable of generalizing to authentic, native-script text. To systematically diagnose this, Amir Hossein Kargaran et al. developed GlotScript: A Resource and Tool for Low Resource Writing System Identification, demonstrating that low-resource language subsets in web-scale crawls suffer from significantly lower script accuracy than their high-resource counterparts.

Why Automated Language Identification Fails

The root cause of this contamination lies in how Language Identification (LID) models operate within web-scraping pipelines. LID tools typically rely on statistical distributions of characters or short text segments (n-grams). While highly effective for massive, homogeneous documents in high-resource languages, this approach breaks down in low-resource settings due to two primary failure modes:

1. The Boilerplate Problem

Web pages written in low-resource languages are rarely purely monolingual. They are frequently embedded within web frameworks that feature English navigation bars, privacy policies, user interface elements, and advertisements.

As observed by Guillaume Wenzek et al. in CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data, this English-language boilerplate heavily interferes with classification. Traditional line-level classifiers often isolate these noisy fragments and misclassify them, or conversely, mistake an entire page of a rare language for English because the boilerplate dominates the character count. The authors of Towards a Cleaner Document-Oriented Multilingual Crawled Corpus argue that shifting from line-level to document-level classification can preserve contextual coherence, yet the tail of rare languages remains highly vulnerable to being overshadowed by global linguistic patterns.

2. The Proximity Trap

LID models struggle significantly when encountering linguistically or orthographically similar languages. When a model is unsure of a text's origin, its underlying probability distribution biases it toward the language it has seen most frequently during training.

This dynamic manifests clearly in regional language clusters. For instance, Leon Derczynski et al. point out in The Danish Gigaword Corpus that automated tools routinely confuse closely related neighboring languages due to overlapping vocabularies and orthography. In lower-resource contexts, this effect is even more damaging. As Amir Hossein Kargaran et al. explain in GlotLID: Language Identification for Low-Resource Languages, LID models frequently misclassify unknown dialects or rare languages as their most prominent high-resource relatives. This systematically injects major-language data into low-resource corpora, diluting the unique grammatical and lexical features the dataset was intended to capture.

The Cost of Representation Washing

The consequence of relying on unverified, automated pipelines is a massive drop in data quality. Manual audits conducted by Ife Adebara et al. in Towards Afrocentric NLP for African Languages: Where We Are and Where We Can Go revealed that some widely used multilingual corpora contain less than 50% correct data for underrepresented languages. The rest consists of mislabeled noise, boilerplate, or foreign text.

This linguistic noise severely degrades downstream model performance. For low-resource languages, raw data volume cannot compensate for poor data purity. Jesujoba O. Alabi et al. demonstrated this trade-off in Massive vs. Curated Embeddings for Low-Resourced Languages: the Case of Yorùbá and Twi, showing that smaller, hand-curated corpora with consistent diacritics and strict language purity vastly outperform massive, web-scraped datasets in training high-quality word representations.

Similarly, when building resources for minority scripts, prioritizing data integrity over sheer scale yields superior results. In MC²: Towards Transparent and Culturally-Aware NLP for Minority Languages in China, Chen Zhang et al. established that filtering out language contamination and ensuring authentic writing systems allows smaller, open-source models to rival much larger closed-source models that suffer from polluted pre-training data.

Ultimately, the literature demonstrates that the hands-off, web-scraping paradigm that succeeded for high-resource languages fails when applied to the long tail of global languages. Building reliable multilingual AI requires moving away from unverified automated pipelines and toward strict script verification, advanced document filtering, and human-in-the-loop curation.

Key Foundational Work

Linguistic profiling of texts for the purpose of language verification (2004) — contributed 1 concept elements to this direction.
Overview of WebCLEF 2006 (2006) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_script_pollution_c190a569,
  title = {script pollution},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/script-pollution/rdr_c190a5696386453590aac980738bc000},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_script_pollution_c190a569_v1,
  title = {script pollution},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/script-pollution/rdr_c190a5696386453590aac980738bc000/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake