Abstract
We present the first French partition of the OLDI Seed Corpus, our submission to the WMT 2025 Open Language Data Initiative (OLDI) shared task. We detail its creation process, which involved using multiple machine translation systems and a custom-built interface for post-editing by qualified native speakers. We also highlight the unique translation challenges presented by the source data, which combines highly technical, encyclopedic terminology with the stylistic irregularities characteristic of user-generated content taken from Wikipedia. This French corpus is not an end in itself, but is intended as a crucial pivot resource to facilitate the collection of parallel corpora for the under-resourced regional languages of France.
Introduction: The Strategic Pivot
Machine Translation (MT) has achieved high performance for major languages like English and French, yet most of the world’s 7,000+ languages remain under-resourced. The Open Language Data Initiative (OLDI) addresses this by providing a "seed" corpus of English Wikipedia sentences to kickstart low-resource MT. However, for regional languages of France—such as Occitan, Picard, or Francoprovençal—English is a linguistically and culturally distant starting point.
Because these regional languages exist in a diglossic relationship with French, a French source text allows for direct calques and terminological consistency that English cannot provide. To bridge this gap, we have developed a gold-standard French version of the OLDI Seed Corpus to act as a strategic pivot resource.
Methodology: Human-in-the-Loop Post-Editing
To ensure high quality while maintaining cost-efficiency, we employed a Machine Translation Post-Editing (MTPE) workflow. Rather than translating 6,000 sentences from scratch, we utilized a "human-in-the-loop" approach involving nine different machine translation configurations:
- Encoder-Decoder Models: NLLB-3.3B, NLLB-600M, and MADLAD-400.
- Large Language Models (LLMs): Llama 4 Scout (109B) and DeepSeek-R1.
- Contextual Variations: We tested segment-level versus document-level translation, including providing models with existing French Wikipedia articles as in-domain context.
We developed a dedicated web interface (Figure 1) that allowed native French experts to compare these nine outputs, select the most accurate hypothesis, and perform final refinements. This process focused on resolving technical terminology and correcting disfluencies inherent in the original English source text.
Key Insights and Model Performance
Our evaluation revealed several critical findings regarding current MT capabilities:
- LLM Proficiency: Researchers were consistently impressed by DeepSeek-R1, which provided "perfect" translations for over 40% of segments. However, its performance was hindered by safety filters that caused it to refuse encyclopedic content related to religion or politics.
- The Context Paradox: Surprisingly, providing models with full Wikipedia articles as context sometimes degraded quality, as models occasionally prioritized matching existing Wikipedia text over accurately translating the specific source segment.
- Technical Terminology: Specialized models like NLLB and MADLAD remained highly competitive, particularly in handling the dense, encyclopedic nature of the Wikipedia-sourced data.
Results
We evaluated the outputs using MetricX-24, a state-of-the-art metric where lower scores indicate higher quality. As expected, human post-editing achieved the superior score of 2.0790.
Among machine systems, NLLB-3.3B and MADLAD-400-3B were the strongest performers. While DeepSeek-R1 showed top-tier potential, its overall average was affected by the aforementioned safety-filter refusals. The workflow successfully demonstrated that MTPE is a viable, high-quality alternative to traditional professional translation for creating foundational "seed" datasets.
Conclusion and Future Work
The French OLDI Seed Corpus serves as a vital resource for the regional languages of France. While we have addressed many of the "noise" issues found in the original English source, future work will involve monitoring the corpus's utility as a pivot and refining terminology as regional translation efforts progress. The dataset and the supplementary hypothesis dataset are available under a CC BY-SA 4.0 license.
Summary updated