Approches en corpus pour la traduction : le cas MÉTÉO

Philippe Langlais, Thomas Leplus, Simona Gandrabur, Guy LapalmeTALN (Articles Courts) 2005 · OpenReview

Abstract: Philippe Langlais, Thomas Leplus, Simona Gandrabur, Guy Lapalme. Actes de la 12ème conférence sur le Traitement Automatique des Langues Naturelles. Articles courts.

2005.

Introduction

While much of the machine translation (MT) research in the mid-2000s focused on broad-domain tasks like news translation, this study investigates a highly specialized, real-world application: the translation of Canadian weather bulletins. Known as the METEO task, this domain was historically dominated by a rule-based system used by Environment Canada. The authors explore whether modern, data-driven techniques can match the performance of this legacy system while offering a more efficient development cycle.

Methodology

The researchers evaluated and combined several corpus-based approaches to handle the specific constraints of weather reports:

  • Phrasal Translation Memory (TM): This system uses edit distance to retrieve the closest matching source phrase from the training data and returns its associated translation. Given the repetitive nature of weather reports, this is a powerful baseline.
  • Phrase-Based Statistical Machine Translation (SMT): The authors utilized the Pharaoh decoder, a state-of-the-art statistical engine at the time, to generate translations based on probabilistic models.
  • Consensus Translation: In a novel cross-disciplinary move, the authors used CLUSTALW—a bioinformatics tool for protein sequence alignment—to align multiple translation candidates and extract a consensus path from the resulting word lattice.
  • Neural Rescoring: A Multi-Layer Perceptron (MLP) was employed to re-rank the n-best lists from the SMT engine. Features for the MLP included source/target length ratios, posterior probabilities, and IBM Model scores.

Ultimately, the authors proposed a hybrid system that prioritizes exact matches from the TM and falls back to the rescored SMT for novel phrases.

Datasets and Evaluation

The study utilized a proprietary English-French bitext from Environment Canada. The data was split chronologically into training, development, and test sets to simulate real-world deployment. A key characteristic of this dataset is its high repetitiveness; approximately 83% of phrases in the development set appeared verbatim in the training data.

Results and Analysis

The experimental results demonstrate the strengths of combining different paradigms. While the simple Translation Memory performed well due to the formulaic nature of the text, the statistical engine proved essential for handling novel sentences.

Neural rescoring provided a significant boost to the pure SMT approach, as shown in the comparison below:

Performance comparison between standard SMT and SMT with neural rescoring.

The most effective configuration was the hybrid system, which combined the reliability of the TM for known phrases with the flexibility of the rescored SMT for unseen data. This system achieved the highest overall performance across all metrics.

Final performance metrics for the hybrid system (TM + Rescored SMT).

Discussion and Conclusion

A critical insight from this work is the limitation of automated metrics like Sentence Error Rate (SER). The authors found that 77% of the "errors" flagged by exact-match metrics were actually valid translations. This highlights the difficulty of evaluating MT in domains where multiple correct paraphrases exist.

By achieving a Word Error Rate (WER) of 4.85% and a BLEU score of 89.59, the authors demonstrated that data-driven methods could indeed rival specialized rule-based systems. This approach offers a faster, more adaptable alternative to manual rule-writing for practical, domain-specific translation tasks.

Summary updated

Abstract

Philippe Langlais, Thomas Leplus, Simona Gandrabur, Guy Lapalme. Actes de la 12ème conférence sur le Traitement Automatique des Langues Naturelles. Articles courts. 2005.
BibTeX
@inproceedings{langlais2005approches,
  author = {Philippe Langlais and Thomas Leplus and Simona Gandrabur and Guy Lapalme},
  title = {{Approches en corpus pour la traduction : le cas MÉTÉO}},
  booktitle = {TALN (Articles Courts)},
  year = {2005},
  url = {https://openreview.net/forum?id=A7Q9ZIeu0Nv},
}
Figure