Findings of the First Shared Task for Creole Language Machine Translation at WMT25

Nathaniel D. Robinson, Claire Bizon Monroc, Rasul Dent, Stefan Watson, Kenton Murray, Raj Dabre, André Coy, Heather Lent2025 · DOI

Abstract

Efforts towards better machine translation (MT) for Creole languages have historically been isolated, due to Creole languages' geographic and linguistic diversity. However, most speakers of Creole languages stand to benefit from improved MT for low-resource languages. To galvanize collaboration for Creole MT across the NLP community, we introduce the First Shared Task for Creole Language Machine Translation at WMT25. This Shared Task consists of two systems tracks and one data track, for which we received submissions from five participating teams. Participants experimented with a wide variety of systems and development techniques. Our evaluation campaign gave rise to improvements in MT performance in several languages, and particularly large improvements in new testing genres, though some participants found that reusing subsets of pretraining data for specialized post-training did not yield significant improvements. Our campaign also yielded new test sets for Mauritian Creole and a vast expansion of public training data for two Creole languages of Latin America.

Introduction: The Creole MT Challenge

Creole languages, spoken by over 180 million people worldwide, emerge from intense contact between different language groups—typically a European "lexifier" and various indigenous or African languages. Despite their significant speaker bases, these languages face extreme data scarcity in the field of Natural Language Processing (NLP).

Research has historically been fragmented because Creole languages are geographically dispersed across the Caribbean, Africa, and the Pacific. Furthermore, they often suffer from social stigma, being incorrectly categorized as "dialects" rather than distinct languages. This shared task aims to create unified benchmarks and state-of-the-art systems for these diverse but typologically related languages.

Geographic distribution of Creole languages included in the shared task, highlighting their global dispersion.

Methodology and Shared Task Tracks

The shared task was organized into three distinct tracks to encourage diverse contributions:

  1. Constrained Track: Participants were limited to the provided KREYÒL-MT dataset. Successful strategies included joint training on typologically related languages (e.g., grouping French-based Creoles) and exploring parameter-efficient fine-tuning like LoRA.
  2. Unconstrained Track: Participants could leverage any external data or Large Language Models (LLMs). A standout approach involved continuous pre-training of Llama 3.1-8B on monolingual Mauritian Creole tokens before fine-tuning for translation.
  3. Data Track: This track focused on the submission of new, high-quality bitexts. Notably, submissions from JHU increased the publicly available bitext for Belizean Kriol by 2,300% and for French Guianese Creole by 370%.

Key Insights and Results

Linguistic Grouping and Model Adaptation

A primary finding was that treating all Creoles as a single category is less effective than grouping them by their lexifier. Joint training on French-based Creoles (such as Mauritian, Seychellois, and Réunion) yielded better results due to shared vocabulary and structural similarities. In the unconstrained track, the most significant performance jumps came from adapting massive pre-trained models like Llama 3.1 to specific Creoles.

Mauritian Creole Success Story

The KOZKREOLMRU system demonstrated the power of LLM adaptation for Mauritian Creole (mfe). By leveraging Llama 3.1, the team achieved substantial improvements over the baseline across multiple test sets, including the new LALIT newspaper benchmark.

Comparison of baseline and primary model performance for Mauritian Creole (mfe) across FLORES and LALIT test sets.

Lusophone Creoles

The EHOW team showed strong performance across seven Portuguese-based Creoles. Their success highlighted that initializing embeddings with tokens from the lexifier (Portuguese) is a highly effective strategy for low-resource translation in these language families.

Discussion and Future Work

While the task achieved significant gains, several challenges remain. Evaluation revealed "contamination" issues where public test sets had leaked into the training data of some popular models, necessitating manual decontamination and re-evaluation. Additionally, most available Creole data is religious or news-based, leaving a dire need for datasets reflecting everyday spoken language.

Future work will focus on expanding genre diversity and developing more human-centric evaluation metrics, as automated metrics like chrF++ and BLEU may not fully capture the nuances of Creole languages, particularly given their high degrees of orthographic variation.

Summary updated

Abstract

Efforts towards better machine translation (MT) for Creole languages have historically been isolated, due to Creole languages' geographic and linguistic diversity.However, most speakers of Creole languages stand to benefit from improved MT for low-resource languages.To galvanize collaboration for Creole MT across the NLP community, we introduce the First Shared Task for Creole Language Machine Translation at WMT25.This Shared Task consists of two systems tracks and one data track, for which we received submissions from five participating teams.Participants experimented with a wide variety of systems and development techniques.Our evaluation campaign gave rise to improvements in MT performance in several languages, and particularly large improvements in new testing genres, though some participants found that reusing subsets of pretraining data for specialized post-training did not yield significant improvements.Our campaign also yielded new test sets for Mauritian Creole and a vast expansion of public training data for two Creole languages of Latin America.
BibTeX
@misc{robinson2025findings,
  author = {Nathaniel D. Robinson and Claire Bizon Monroc and Rasul Dent and Stefan Watson and Kenton Murray and Raj Dabre and André Coy and Heather Lent},
  title = {{Findings of the First Shared Task for Creole Language Machine Translation at WMT25}},
  year = {2025},
  doi = {10.18653/v1/2025.wmt-1.28},
}
Figure