MC^2: A Multilingual Corpus of Minority Languages in China

Chen Zhang, Mingxu Tao, Quzhe Huang, Jiuheng Lin, Zhibin Chen, Yansong Feng

Abstract

Large Language Models (LLMs) frequently underperform on low-resource languages due to data scarcity and poor data quality. We present MC², the largest open-source multilingual corpus for four underrepresented minority languages in China: Tibetan, Uyghur, Kazakh, and Mongolian. By emphasizing high-quality curation and script-specific data (e.g., Traditional Mongolian and Arabic-script Kazakh), the MC² corpus enables models to outperform existing benchmarks and better reflect the cultural nuances of these linguistic communities.

Addressing the Data Gap in Minority Languages

While LLMs excel in high-resource languages like English and Chinese, they face a significant "digital divide" when applied to minority languages. The researchers identify three primary challenges:

  1. Extreme Data Scarcity: There is a massive disparity between the population size of these language speakers and the amount of available tokens in existing datasets like CulturaX.
  2. Data Contamination: Multilingual datasets often misidentify text; for example, up to 34% of "Uyghur" data in some popular sets is actually Kazakh or Arabic.
  3. Script Neglect: International datasets often favor Cyrillic scripts for Kazakh and Mongolian, ignoring the Arabic and Traditional scripts used by millions in China. This neglect is not just a technical hurdle but a form of cultural erasure.
Figure 1: Comparison of population size vs. available tokens in the CulturaX dataset, highlighting the severe underrepresentation of minority languages like Tibetan (bo), Uyghur (ug), Kazakh (kk), and Mongolian (mn).

The MC² Corpus: Quality Over Quantity

The researchers developed a quality-centric pipeline to build the MC² corpus, moving away from uncurated web-scale scraping in favor of precision.

Methodology

  • Curated Crawling: The team maintained a whitelist of high-quality websites for each language to ensure linguistic purity.
  • AI-Assisted Extraction: Using GitHub Copilot, they created site-specific scripts to extract main content while discarding non-semantic "noise" like headers, footers, and sidebars.
  • Continual Pre-training: They performed continual pre-training on XLM-RoBERTa and Llama 2, expanding the vocabulary by 3,000 tokens per language to accommodate unique scripts.

Dataset Composition

The resulting corpus provides substantial data for previously neglected scripts, totaling several gigabytes of high-quality text across Tibetan, Uyghur, Kazakh (Arabic script), and Mongolian (Traditional script). As shown in the distribution below, the corpus relies heavily on professional sources to maintain standards.

Figure 5: Distribution of data sources in the MC² corpus, showing a heavy reliance on News and Government documents to ensure high linguistic quality.

Key Insights: Script as a Cultural Vessel

A pivotal finding of this research is that scripts encapsulate culture. Probing experiments revealed that a Kazakh model trained on Cyrillic script (Kazakhstan-centric) associated national holidays and currency with Kazakhstan, while an Arabic-script model (China-centric) correctly identified Chinese cultural markers like the Renminbi. This demonstrates that authentic, script-specific data is irreplaceable for capturing a language's unique "worldview."

Furthermore, the study showed that "cleaning dividends" are real: models trained on the smaller but cleaner MC² corpus achieved performance comparable to or better than models trained on much larger, noisier datasets. Authentic data is more valuable than raw volume.

Performance Results

  • MC²XLMR-large: Outperformed vanilla XLM-RoBERTa, achieving a weighted F1 score of 48.9 on classification tasks, rivaling the closed-source CINO model despite using significantly less data.
  • MC²Llama-13B: Showed a 26% accuracy improvement over other open-source LLMs (like BLOOM and mT5) in few-shot classification benchmarks.
  • Zero-Shot Transfer: The models demonstrated strong cross-lingual transfer from high-resource languages like Chinese to minority languages, particularly in Question Answering tasks.

Limitations and Future Directions

The authors note that the corpus is currently biased toward news and government domains due to the availability of high-quality web sources. Future work will focus on incorporating digitized books to diversify linguistic styles and expanding cultural probing benchmarks beyond the initial 25-question set. Additionally, the presence of "inter-sentential code-switching" in the data suggests potential for mining parallel translation datasets in the future.

Summary updated

BibTeX
@inproceedings{zhang2024mc,
  author = {Chen Zhang and Mingxu Tao and Quzhe Huang and Jiuheng Lin and Zhibin Chen and Yansong Feng},
  title = {{MC^2: A Multilingual Corpus of Minority Languages in China}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=YZ1dAvlGqN},
}
Figure