The TALP-UPC System for the WMT Similar Language Task: Statistical vs Neural Machine Translation

Magdalena Biesialska, Lluis Guardia, Marta R. Costa-jussàACL 2019

TL;DR

This paper investigates the performance of statistical (phrase-based) and neural machine translation (NMT) systems on the challenging task of translating between similar languages. Surprisingly, both approaches yielded comparable results, with performance varying by language pair, and the authors' system achieved top rankings in the WMT Similar Language Translation task.

What Problem Does This Paper Address?

Translating between similar languages, like those within the same linguistic family, presents unique challenges. While one might assume these languages are easier to translate due to shared vocabulary and grammar, they can also have subtle but significant differences. These can include variations in alphabets (e.g., diacritics), which can trip up MT systems. This paper tackles the question of whether traditional statistical machine translation (SMT) or modern neural machine translation (NMT) is better suited for this specific task, especially when resources are limited. They also explore how to best leverage monolingual data, which is often more abundant than parallel data, to improve translation quality.

Methods

The authors compare two primary machine translation paradigms:

  • Statistical Machine Translation (SMT): Specifically, they use a phrase-based SMT system (Moses). This approach breaks down sentences into phrases, translates them individually, and then reorders them. It relies on a translation model built from aligned parallel corpora and a language model trained on monolingual target language data.
  • Neural Machine Translation (NMT): They employ the Transformer architecture, a state-of-the-art NMT model that relies solely on self-attention mechanisms, eschewing recurrence and convolution. This allows it to model dependencies between all words in a sentence simultaneously.

They also investigate strategies for incorporating monolingual data:

  • Monolingual Data: Directly using monolingual data in the target language to improve fluency.
  • Back-translation: Translating monolingual target data back into the source language to create synthetic parallel data for training.
  • System Combination: Combining the outputs of their SMT and NMT systems using Minimum Bayes Risk (MBR) and a novel back-translation-based combination strategy.

Datasets

The experiments were conducted using datasets provided by the organizers of the 1st WMT Similar Language Translation task.

  • Czech-Polish: Approximately 2.2 million parallel sentences and 1.2 million monolingual Polish sentences.
  • Spanish-Portuguese: Approximately 4.5 million parallel sentences and 10.9 million monolingual Portuguese sentences.

Development and test sets were created from the provided data.

Key Insights and Ideas

  • Similar Languages, Similar Performance: Contrary to expectations, statistical and neural MT systems performed comparably on similar language pairs. This suggests that the "similarity" doesn't automatically favor one approach over the other, and the specific linguistic differences (like alphabet variations) play a crucial role.
  • Language Pair Matters: The performance gap between SMT and NMT varied significantly. SMT outperformed NMT by 6 BLEU points for Spanish-Portuguese (a more similar pair), while NMT surpassed SMT by 2 BLEU points for Czech-Polish (a less similar pair with alphabet differences).
  • Monolingual Data Can Hurt: Surprisingly, directly incorporating monolingual data into the NMT training process harmed performance, even for similar languages. This highlights the complexity of effectively leveraging such data.
  • Back-translation's Limited Utility: The authors found that their system combination strategy using back-translation did not yield significant improvements. They attribute this to a weak correlation between the quality of back-translations and the actual translation quality, suggesting back-translation might not be a reliable quality estimation metric in this context.
  • Alphabet Differences are a Hurdle: The paper points to the distinct alphabets of Czech and Polish (despite being Slavic languages) as a potential reason for lower BLEU scores, suggesting that character-level differences can be a significant challenge for MT systems.

Key Results

  • The TALP-UPC system achieved 1st place for Czech-Polish and 2nd place for Spanish-Portuguese in the official evaluation of the 1st WMT Similar Language Translation task.
  • For Spanish-Portuguese, SMT achieved a BLEU score of 64.96, while NMT achieved 58.40.
  • For Czech-Polish, NMT achieved a BLEU score of 11.69, while SMT achieved 9.87.
  • System combinations (MBR or back-translation) did not significantly improve upon the individual best systems.

Limitations and Future Work

  • Limited Data for NMT: Due to time constraints, the authors were unable to fully train their NMT model with pseudo-parallel corpora for the Spanish-Portuguese pair.
  • Back-translation Quality Estimation: The authors identified a weak correlation between back-translation quality and actual translation quality, limiting the effectiveness of their combination strategy.
  • Future Work: The authors plan to explore:
  • The impact of removing diacritics from the source text for Czech-Polish translation.
  • The effectiveness of various quality estimation approaches for system combination, including reward scores and the OpenKiwi toolkit.

Summary updated

BibTeX
@inproceedings{biesialska2019talp,
  author = {Magdalena Biesialska and Lluis Guardia and Marta R. Costa-jussà},
  title = {{The TALP-UPC System for the WMT Similar Language Task: Statistical vs Neural Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure