TL;DR
The WMT 2024 Shared Task on the Open Language Data Initiative focused on expanding foundational multilingual datasets, FLORES+ and MT Seed, to support under-resourced languages. Ten submissions successfully added data for 16 languages, improving dataset quality and coverage, and fostering community-driven language technology development.
What Problem Does This Paper Address?
Machine translation (MT) has seen rapid advancements, but progress has largely benefited high-resource languages. While efforts are shifting towards under-resourced languages, a critical bottleneck remains the availability of high-quality, comprehensive evaluation benchmarks and training data. Creating these foundational datasets is crucial for developing and evaluating language technologies for a wider range of languages, bridging the gap between well-resourced and under-resourced linguistic communities. This paper details the WMT 2024 shared task aimed at addressing this by soliciting contributions to expand these vital resources.
Methods
The WMT 2024 Shared Task solicited contributions to two key datasets: FLORES+ (for evaluation) and MT Seed (for training). Participants were encouraged to translate from English into target languages, following established guidelines. For FLORES+, contributions had to be entirely human-translated, with no machine translation (MT) output or post-editing allowed, to ensure high-quality evaluation data. For MT Seed, post-edited MT content was permissible, provided it was manually verified, acknowledging its role as a training resource. Submissions were required to adhere to standardized language codes and include a dataset card detailing their creation and characteristics. All contributions were mandated to be released under open licenses.
Datasets
- FLORES+: An extension of the FLORES family of datasets, managed by OLDI. It consists of sentences extracted from English Wikinews, Wikijunior, and Wikivoyage, professionally translated into various languages. FLORES+ is intended for evaluating MT systems and requires human-translated contributions.
- MT Seed: Previously known as NLLB Seed, this dataset provides "starter data" for languages lacking sufficient high-quality bitext for training NLP models. It comprises sentences sampled from Wikipedia articles, professionally translated into multiple languages. MT Seed is intended for training and allows for post-edited MT contributions.
- New Monolingual Datasets: The task also accepted other high-quality, human-verified monolingual text datasets in under-resourced languages.
Key Insights and Ideas
- Community-Centric Development: The shared task emphasized a community-driven approach, actively engaging native speakers and linguistic experts to ensure the quality and cultural appropriateness of the contributed data.
- The Imperative of Foundational Data: The paper highlights that despite advances in large-scale MT models, the continued development of high-quality, open-access evaluation and training datasets remains critical for equitable progress in language technologies.
- Bridging the Resource Gap: By expanding datasets like FLORES+ and MT Seed, the initiative directly tackles the disparity in resources between high- and low-resource languages, aiming to prevent the exacerbation of technical divides.
- Rigorous Quality Control: The distinction between FLORES+ (human-only translation) and MT Seed (allowing verified post-edited MT) demonstrates a nuanced approach to data creation, balancing evaluation rigor with training data accessibility.
Key Results
The WMT 2024 OLDI shared task accepted ten submissions, contributing data for 16 languages. These contributions extended the coverage of the FLORES+ and MT Seed datasets and improved the quality of existing data. Specific contributions included:
- Corrections and improvements to FLORES+ datasets for Hausa, Northern Sotho, Xitsonga, and isiZulu, addressing issues like poor named entity translation and inconsistency.
- Translations of MT Seed into Bangla/Bengali, which demonstrated strong performance when used to fine-tune pre-trained multilingual models.
- New FLORES+ translations into Emakhuwa, with participants also benchmarking MT progress for this language pair.
- A translation of MT Seed into Italian, which was verified by native speakers and used to train an MT system.
- New FLORES+ translations into Erzya, with the creation of neologisms to aid future translators.
- A FLORES+ devtest split translation into Karakalpak, alongside a large training dataset for Uzbek-Karakalpak, Russian-Karakalpak, and English-Karakalpak.
- New FLORES+ translations into four Romance languages of Spain (Aragonese, Aranese, Valencian) and a corrected Asturian dataset, used in a major shared task.
- A FLORES+ translation into the Chongming dialect of Wu Chinese, which involved careful consideration of colloquialisms and community input.
Limitations and Future Work
- Limitations: The paper notes that while the task aimed to expand data for under-served languages, the overall coverage is still limited. The heterogeneity of submissions meant that the exact nature of experimental validation was left to participants, leading to varied evaluation approaches.
- Future Work: The authors express hope that the showcased work will inspire further efforts to consolidate building blocks for advancing language technologies for under-served linguistic communities. They encourage future work to build upon these foundational datasets. The development of tools like Seed-CAT (mentioned in the submission by Cols (2024)) also points towards future directions in facilitating translator workflows.
Summary updated