The ADAPT System Description for the STAPLE 2020 English-to-Portuguese Translation Task

Rejwanul Haque, Yasmin Moslem, Andy WayACL 2020

TL;DR

This paper describes the ADAPT Centre's submission to the STAPLE 2020 shared task, focusing on generating a high-coverage set of plausible English-to-Portuguese translations. They achieved this by combining multiple translation systems and leveraging techniques like monolingual MT and careful data selection, ultimately earning fifth place in the competition.

What Problem Does This Paper Address?

The STAPLE 2020 shared task aimed to address a specific use case of machine translation (MT): generating a large number of acceptable translations for a given input sentence. This is particularly relevant for language learning platforms like Duolingo, where learners' responses are compared against a diverse set of human-curated translations. The challenge lies in producing not just accurate translations, but a broad spectrum of plausible alternatives, which is more complex than standard MT tasks that typically focus on a single best translation.

Methods

The ADAPT team employed a multi-faceted approach to tackle this challenge:

  • Data Selection: They prioritized external parallel corpora that were similar to the Duolingo training data, using perplexity on in-domain language models to measure similarity.
  • "Pseudo In-domain" Data: They utilized the Axelrod et al. (2011) method to extract sentence pairs from large external corpora that exhibited low cross-entropy differences, effectively creating "pseudo in-domain" data that mimicked the style and domain of the target task.
  • Monolingual MT: They developed Portuguese-to-Portuguese MT systems. This allowed them to generate multiple alternative Portuguese translations for a given input, which were then used to augment their training data.
  • Ensembling and Combination: The core of their strategy involved combining the outputs of multiple MT systems. They experimented with various methods to merge n-best lists from different phrase-based statistical MT (PB-SMT) and neural MT (NMT) models, aiming to maximize coverage and precision. They also explored using monolingual MT systems to further refine and expand the translation sets.

Datasets

  • STAPLE 2020 Training Data: Provided by Duolingo, containing 4,000 English prompts with multiple Portuguese translations, resulting in 526,467 source-target segment-pairs.
  • Development and Test Sets: Derived from the STAPLE 2020 data. A random sample of 2,000 pairs formed the development set, and 500 prompts with high-quality reference translations from Amazon Translate constituted the test set.
  • External Parallel Corpora: A variety of publicly available corpora were used, including OPUS (Tatoeba, Books, OpenSubtitles, ParaCrawl, Wikipedia, EUconst, JRC-Acquis, Europarl, DGT).
  • Portuguese Paraphrase Database (PPDB): Used for augmenting the Portuguese-to-Portuguese MT training data.

Key Insights and Ideas

  • Data Similarity is Key: The effectiveness of external data was significantly boosted by selecting corpora that closely matched the domain and style of the target task (Duolingo data).
  • Monolingual MT for Diversity: Using monolingual MT to generate alternative translations for the target language proved to be a valuable strategy for increasing the diversity of the final output sets.
  • Ensembling for Coverage: Combining the outputs of multiple MT systems, especially those trained on different data sources or using different architectures, was crucial for achieving high coverage of plausible translations.
  • Trade-off between Precision and Recall: The paper highlights that while combining systems increased recall (coverage), it sometimes came at the expense of precision, a common challenge in generating diverse outputs.
  • PB-SMT vs. NMT Performance: Interestingly, in some low-resource scenarios and for specific tasks (like monolingual MT), PB-SMT systems showed competitive or even superior performance to NMT systems.

Key Results

  • The ADAPT team's best system, NEURAL-BEST, achieved an F₁ score of 0.4601 on the development set.
  • Their final submitted system, a combination of multiple NMT and PB-SMT models with monolingual MT augmentation, secured fifth place in the STAPLE 2020 competition.
  • The best-performing PB-SMT system (PB-BEST) showed significant improvements over the baseline, achieving a BLEU score of 49.42 on the test set when trained on augmented data.
  • The monolingual Portuguese MT systems achieved high BLEU scores (72.30 for PB-SMT, 40.73 for NMT), indicating their effectiveness in generating variations of Portuguese sentences.

Limitations and Future Work

  • Identical Translations: A limitation identified in the PB-BEST system was the generation of many identical translations, leading to low recall.
  • Future Work: The authors plan to explore confusion network decoding for re-ranking n-best translations and to apply monolingual MT to the source (English) side for generating alternative input sentences. They also aim to refine the combination strategy to better balance precision and recall.

Summary updated

BibTeX
@inproceedings{haque2020adapt,
  author = {Rejwanul Haque and Yasmin Moslem and Andy Way},
  title = {{The ADAPT System Description for the STAPLE 2020 English-to-Portuguese Translation Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure