Data Curation and Adaptation for Underrepresented Languages

77 concepts76 papers1 proposal

Modern natural language processing relies on an implicit assumption: the availability of web-scale text. Massive crawls of the internet provide the trillions of tokens necessary to train large language models (LLMs). However, this reliance creates a profound digital language divide. As highlighted by Alham Fikri Aji in One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia, a massive disparity exists between a language's speaker population and its digital footprint. Languages spoken by tens or hundreds of millions of people often find themselves in a "data desert" because they are underrepresented in standard web crawls.

To bridge this gap, researchers are shifting from passive web scraping to active, targeted data curation and innovative model adaptation strategies. The goal is not merely to collect more tokens, but to maximize the utility of every available token through specialized architectures, community engagement, and cross-lingual transfer.

From Data Mining to Data Farming

The traditional approach to building pretraining corpora relies on automated web scraping. However, for low-resource languages, this often yields noisy, low-quality data or fails entirely due to inaccurate language identification—a problem explored in the context of Nordic languages by the authors of Discriminating Between Similar Nordic Languages.

To counteract this, recent initiatives favor "data farming"—the active, community-driven cultivation of high-quality language resources. For instance, Chris Chinenye Emezue et al. introduced The NaijaVoices Dataset: Cultivating Large-Scale, High-Quality, Culturally-Rich Speech Data for African Languages, leveraging a community-centric approach to gather over 1,800 hours of culturally rich speech data for Nigerian languages. Similarly, Devansh Mehta et al. deployed localized interventions, such as children's book translation workshops and gamified crowdsourcing, to bootstrap resources for the Gondi language in Learnings from Technological Interventions in a Low Resource Language: A Case-Study on Gondi.

These localized efforts are being scaled into massive regional hubs that standardize evaluation and data access. Divyanshu Kakwani et al. developed iNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages, providing an 8.8-billion-token corpus and standardized benchmarks for 11 major Indian languages. On an even larger collaborative scale, Holy Lovenia et al. established SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages to systematically map and benchmark the linguistic diversity of Southeast Asia, demonstrating that structured, human-verified data hubs are essential for overcoming web-representation biases.

The Power of Specialized Monolingual Models

When data is scarce, a common engineering shortcut is to rely on massive, general-purpose multilingual models like mBERT or XLM-R. However, these models suffer from the "curse of multilinguality": their vocabulary and parameter capacity are diluted across hundreds of languages, often leading to poor performance on nuanced regional tasks.

Intuition suggests that a smaller model trained exclusively on a high-quality, targeted corpus can outperform a massive model that only encounters the target language occasionally. This is validated by Alistair Plum in Sinhala Encoder-only Language Models and Evaluation, showing that dedicated monolingual models significantly outperform popular multilingual giants because their internal representations are entirely optimized for the target language's syntax and morphology.

Furthermore, data transparency and quality can compensate for a lack of scale. Gian Carlos Tan et al. demonstrated this with CebBERT: A Lightweight Data-Transparent DistilBERT Model for Cebuano Language Processing. By prioritizing a diverse, high-quality corpus over sheer volume, they built a model 40% smaller and 60% faster than standard BERT that still excelled at complex semantic tasks. Even when scaling up to larger architectures, tailoring the model to a specific language's constraints remains vital, as shown by Risto Luukkonen et al. in FinGPT: Large Generative Models for a Small Language, which optimized large generative models specifically for the structural and data limitations of the Finnish language.

Bootstrapping via Cross-Lingual Transfer and Adaptation

What happens when a language has virtually no digital text available? In these extreme low-resource scenarios, researchers bootstrap performance by transferring knowledge from higher-resource languages or utilizing structural linguistic priors.

One approach is to directly map the representations of a high-resource model into a new language space using minimal parallel data. Qingcheng Zeng et al. introduced GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Cost, which uses bilingual lexicons to translate an English model's vocabulary space into a target language. This initialization strategy allows the model to inherit downstream reasoning capabilities, outperforming from-scratch training while using up to 200x less data and compute.

Another strategy leverages the mutual intelligibility between closely related languages or dialects. The authors of KINNEWS and KIRNEWS: Benchmarking Cross-Lingual Text Classification for Kinyarwanda and Kirundi demonstrated that text classification models trained on the relatively higher-resource Kinyarwanda language could successfully generalize to Kirundi without requiring expensive parallel corpora, proving that linguistic proximity can serve as a proxy for raw data volume.

Finally, structural linguistic knowledge can be injected directly into modern prompting paradigms without any parameter updates. Rita Ramos developed GrammaMT: Improving Machine Translation with Grammar-Informed In-Context Learning, a training-free strategy that provides LLMs with Interlinear Glossed Text (IGT). By embedding explicit morphological and lexical glosses into the prompt, GrammaMT provides the structural scaffolding necessary for the model to translate endangered and low-resource languages accurately, bypassing the need for massive parallel training corpora.

Key Foundational Work

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (2005) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_low_resource_language_data_scarcity_30ed5c35,
  title = {low-resource language data scarcity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/low-resource-language-data-scarcity/rdr_30ed5c355c7040019cf99d8c5293c524},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_low_resource_language_data_scarcity_30ed5c35_v1,
  title = {low-resource language data scarcity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/low-resource-language-data-scarcity/rdr_30ed5c355c7040019cf99d8c5293c524/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake