Proceedings of the First Workshop on Language Models for Low-Resource Languages

Marjana Prifti Skenduli, Hansi Hettiarachchi, Tharindu Ranasinghe, Paul Rayson, Ruslan Mitkov, Mohamed Medhat Gaber, Damith Premasiri, Fiona Anting Tan, Lasitha UyangodageWorkshops @ COLING 2025

Introduction

Modern Natural Language Processing (NLP) is currently characterized by a "rich-get-richer" cycle. While neural language models like GPT-4 and Llama have achieved remarkable milestones, their capabilities are primarily defined by the massive English-centric corpora used during training. This leaves the vast majority of the world’s ~7,000 languages—many with millions of speakers—in a "low-resource" state, lacking the digital data, benchmarks, and research attention needed to benefit from AI advancements.

The First Workshop on Language Models for Low-Resource Languages (LoResLM 2025), held at COLING 2025, addresses this "digital divide." The workshop presents 35 new studies aimed at dismantling high-resource bias by introducing novel datasets, data-efficient training methods, and cultural benchmarks for 28 underserved languages. The goal is to establish a roadmap for global linguistic fairness in the era of Large Language Models (LLMs).

Methodologies for Resource-Constrained Environments

The proceedings highlight a shift from "brute force" scaling to clever, resource-conscious engineering. Key methodologies discussed include:

  • Language-Adaptive Fine-Tuning (LAFT): Adapting existing multilingual models (e.g., AfriBERTa) to specific languages such as Hausa.
  • Knowledge Distillation: Techniques to "distill" the knowledge of massive transformers into smaller, general-use models capable of running in resource-constrained environments.
  • Adaptive Negative Sampling: A novel approach to training that prevents "token marginalization," where rare language tokens are ignored by standard loss functions.
  • Morphology-Enhanced Modeling: Incorporating specific linguistic structures (e.g., for Ancient Greek or isiXhosa) into neural architectures to compensate for raw data scarcity.
  • Reference-less Evaluation: Developing methods to judge translation quality when "gold standard" human translations are unavailable.

New Benchmarks and Datasets

A core contribution of the workshop is the release of several new benchmarks designed to fill specific gaps:

  • BnSentMix: A Bengali-English code-mixed dataset for sentiment analysis.
  • Atlas-Chat: A dataset for adapting LLMs to the Moroccan Arabic dialect.
  • PersianMCQ-Instruct: A resource for generating multiple-choice questions in Persian.
  • CaLQuest.PT: A collection of "Causal Ladder" questions in Portuguese for evaluating AI reasoning.
  • Cultural Benchmarks: New frameworks for measuring sexist and homophobic bias in Southeast Asian languages and a culturally aligned benchmark for Arabic LLMs.

Key Insights: Beyond Data Quantity

The workshop challenged the "more is always better" dogma through several key insights:

  • The Math of Marginalization: Standard training objectives like Cross-Entropy loss may inherently disadvantage low-resource languages. Rare tokens often "jostle" for space in a model's vocabulary, requiring techniques like Adaptive Negative Sampling to ensure they are properly learned.
  • Data Efficiency: The "BabyLMs" project (specifically for isiXhosa) demonstrates that models can be surprisingly effective even when trained on very small, high-quality datasets.
  • Linguistic Fairness: The workshop frames NLP development as a matter of "linguistic fairness," arguing that cultural alignment is as vital as grammatical accuracy.
  • Historical Continuity: "Low-resource" status extends to historical languages like Medieval Latin and Ancient Greek, which require specialized "continuous training" to adapt modern BERT models.

Results and Representation

The workshop successfully curated a diverse body of work covering 28 low-resource languages across 8 language families. While Indo-European languages still accounted for 47.2% of the studies, there were significant contributions from African, Turkic, and Southeast Asian linguistic groups. Research spanned 13 areas, with the most activity in Language Modelling, Machine Translation, and Lexical Semantics. Technical successes included BERT-based part-of-speech tagging for Uzbek (BBPOS) and improved idiomatic translation in LLMs.

Limitations and Future Directions

Despite progress, challenges remain. Nearly half of the research still focuses on Indo-European branches, highlighting a need for even broader diversity. Furthermore, there is a persistent "evaluation gap"—a lack of robust, automated ways to evaluate LLMs in low-resource settings without relying on English-based metrics or expensive human annotation. Future work must bridge the gap between laboratory results and real-world deployment, such as creating conversational agents for informal business management in Africa.

Summary updated

BibTeX
@inproceedings{skenduli2025proceedings,
  author = {Marjana Prifti Skenduli and Hansi Hettiarachchi and Tharindu Ranasinghe and Paul Rayson and Ruslan Mitkov and Mohamed Medhat Gaber and Damith Premasiri and Fiona Anting Tan and Lasitha Uyangodage},
  title = {{Proceedings of the First Workshop on Language Models for Low-Resource Languages}},
  booktitle = {Workshops @ COLING},
  year = {2025},
}
Figure