The MLLP-UPV Spanish-Portuguese and Portuguese-Spanish Machine Translation Systems for WMT19 Similar Language Translation Task

Pau Baquero-Arnal, Javier Iranzo-Sánchez, Jorge Civera, Alfons JuanACL 2019

TL;DR

This paper introduces a novel "2D alternating RNN" architecture for machine translation, which treats translation as a two-dimensional problem. While a Transformer baseline achieved state-of-the-art results after domain adaptation, the new RNN architecture shows competitive performance with significantly fewer parameters, hinting at future potential.

What Problem Does This Paper Address?

Machine translation (MT) systems, particularly Neural Machine Translation (NMT), have seen remarkable progress. However, a persistent challenge is achieving high-quality translation when the training data (general domain) doesn't perfectly match the target domain (specific domain). This mismatch can lead to performance degradation. The WMT19 Similar Language Translation Shared Task provided an opportunity to test systems on closely related languages (Spanish and Portuguese) and to explore domain adaptation techniques. The authors aim to develop and evaluate NMT systems for Spanish-Portuguese and Portuguese-Spanish translation, introducing a new architecture and demonstrating the effectiveness of domain adaptation.

Methods

The MLLP-UPV research group developed two main types of NMT systems:

  • Transformer Baseline: They used the well-established Transformer architecture (Vaswani et al., 2017), employing a "Base" configuration. This involved standard practices like gradient accumulation for larger effective batch sizes, dropout, label smoothing, and the Adam optimizer.
  • 2D Alternating RNN: This is the paper's novel contribution. It's a new architecture that views translation as a two-dimensional problem, where each target word is explained by all source words. It leverages recurrent cells (like GRU) and processes information in a grid-like structure, considering both source and target dimensions simultaneously. This is a departure from traditional encoder-decoder models that process them separately.
  • Domain Adaptation (Fine-tuning): To address the domain mismatch issue, they employed a fine-tuning technique. This involves taking a model pre-trained on general domain data and further training it on a small amount of in-domain data (from the development set of the shared task). This process was monitored to prevent overfitting.

Datasets

  • Training Data: The systems were trained on a combination of standard parallel corpora: JCR, Europarl, news-commentary, and wikititles.
  • Development and Test Data: The shared task provided development and test sets for both Portuguese-Spanish and Spanish-Portuguese language pairs. The development data was split into two parts, one for development and one for testing the fine-tuning process.

Key Insights and Ideas

  • Translation as a 2D Problem: The core idea behind the 2D alternating RNN is that translation isn't just a sequential mapping but a more complex interaction across the entire source and target sequences. This perspective could lead to more nuanced understanding.
  • Parameter Efficiency: The 2D alternating RNN achieved competitive results with significantly fewer parameters (14.9M) compared to the Transformer baseline (60.2M). This suggests that architectural innovation can lead to comparable performance without the need for massive model sizes.
  • Domain Adaptation is Crucial: The paper strongly emphasizes that fine-tuning on in-domain data provides substantial performance gains, especially when there's a domain mismatch between training and test sets. This is a critical takeaway for building practical MT systems.
  • Early Fine-tuning Epochs are Most Impactful: The analysis of fine-tuning epochs shows that the most significant improvements occur in the initial stages, with diminishing returns as training continues. This helps in efficiently deciding how long to fine-tune.

Key Results

  • Transformer Baseline: Achieved strong results, with 57.4 BLEU for Portuguese-Spanish and 51.2 BLEU for Spanish-Portuguese on the test set.
  • 2D Alternating RNN Baseline: Showed competitive performance, scoring 55.1 BLEU for Portuguese-Spanish, only slightly below the Transformer baseline, despite having fewer parameters.
  • Fine-tuned Transformer: Achieved state-of-the-art results in the shared task. For Portuguese-Spanish, it reached 72.4 BLEU (a 15.0 BLEU improvement over the baseline). For Spanish-Portuguese, it reached 70.7 BLEU (a 19.4 BLEU improvement).
  • Fine-tuned 2D Alternating RNN: Also benefited significantly from fine-tuning, achieving 64.0 BLEU for Portuguese-Spanish (an 8.9 BLEU improvement).
  • Outperformance in Shared Task: The MLLP-UPV fine-tuned Transformer systems were the top submissions in both language directions of the WMT19 Similar Language Translation Shared Task, outperforming other participants by a significant margin.

Limitations and Future Work

  • 2D RNN Development Stage: The 2D alternating RNN is described as being in early stages of development. It was only trained for one direction (Portuguese-Spanish) due to time constraints.
  • Future Work for 2D RNN: The authors plan to fully develop the 2D alternating RNN by supporting larger embedding/hidden unit sizes, deeper architectures, and more regularization. They believe these improvements will further boost its performance.
  • Domain Mismatch Explanation: While the paper demonstrates the effectiveness of fine-tuning due to domain mismatch, the authors suggest that the success might be more related to the general domain mismatch rather than the specific similarity between Spanish and Portuguese.

Summary updated

BibTeX
@inproceedings{baqueroarnal2019mllp,
  author = {Pau Baquero-Arnal and Javier Iranzo-Sánchez and Jorge Civera and Alfons Juan},
  title = {{The MLLP-UPV Spanish-Portuguese and Portuguese-Spanish Machine Translation Systems for WMT19 Similar Language Translation Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure