Combining Translation Memory with Neural Machine Translation.

Akiko Eriguchi, Spencer Rarrick, Hitokazu MatsushitaWAT @ EMNLP 2019 · OpenReview

TL;DR

This paper presents a system that combines traditional Translation Memory (TM) with Neural Machine Translation (NMT) to improve translation quality, especially for repetitive or known content. By intelligently choosing between TM retrieval and NMT generation based on sentence similarity, the system significantly boosts performance on a specialized corpus, bridging the gap with commercial MT systems.

What Problem Does This Paper Address?

Automatic translation systems, particularly Neural Machine Translation (NMT), have made incredible strides. However, NMT models can struggle with highly repetitive content or domain-specific terminology that might already exist in a company's translation archives. Manually managing these archives (Translation Memories or TMs) is time-consuming, and simply relying on NMT can lead to inconsistencies or suboptimal translations for known phrases. The challenge is to effectively leverage existing, high-quality translations from TMs while still benefiting from the flexibility and fluency of NMT for novel content. This paper addresses how to seamlessly integrate TM retrieval with NMT to achieve the best of both worlds.

Methods

The core of the proposed system is a hybrid approach that intelligently switches between two translation sources: a Translation Memory (TM) and a Neural Machine Translation (NMT) model.

  1. Translation Memory (TM) as a First Pass: The system uses the entire training dataset as a TM. For each incoming source sentence, it calculates a similarity score against all source sentences in the TM.
  2. Retrieval Mechanisms: Two methods are employed for calculating similarity:
  3. Edit-Distance Based Retrieval: Uses the Levenshtein distance (character-based) to measure how similar two sentences are.
  4. IDF-Based Retrieval: Leverages Inverse Document Frequency (IDF) scores of tokens within sentences to gauge similarity, considering word importance.
  5. Conditional NMT Generation: If the similarity score between the source sentence and any TM entry exceeds a predefined threshold, the system retrieves the corresponding target sentence from the TM and uses it as the translation. If no TM entry meets the similarity threshold, the sentence is passed to a standard NMT model (a Transformer-based architecture) for translation.
  6. NMT Model: A Transformer (base) model, similar to the one described in Vaswani et al. (2017), is used. It features a standard encoder-decoder architecture with self-attention mechanisms, positional embeddings, and label smoothing for regularization. The authors experimented with two specific implementations: one using the Marian framework and another using Fairseq.

Datasets

  • Timely Disclosure Documents Corpus: This is the primary dataset used for the WAT (Workshop on Asian Translation) task. It consists of past years' timely disclosure documents, providing approximately 1.4 million Japanese-English sentence pairs. The corpus exhibits a significant number of duplicate training examples.
  • 1M Japanese-English Wikipedia Parallel Corpus: This corpus was used as an additional training resource to augment the main dataset. It was created by crawling and aligning Wikipedia pages.

The evaluation was performed on the development (Dev), devtest, and test sets of this corpus.

Key Insights and Ideas

  • Leveraging Duplication: The paper's motivation stems from the observation that a significant portion (26-28%) of the evaluation data consists of exact duplicates of sentences found in the training data. This high degree of repetition makes TM retrieval a highly relevant and potentially very effective strategy.
  • Task-Specific TM Effectiveness: The authors found that TM retrieval works better for the "ITEM" dataset (which likely contains shorter, more repetitive sentences) than for the "TEXT" dataset (which likely contains longer, more varied sentences). This suggests that TM is particularly well-suited for translating fixed phrases or known sentence structures.
  • Bridging the Gap: The hybrid TM+NMT approach significantly narrows the performance gap between the authors' in-house NMT systems and commercial black-box MT systems, suggesting that TM can compensate for limitations in NMT's ability to handle repetitive content.
  • Retrieval Method Choice Matters: The optimal retrieval method (edit-distance vs. IDF-based) depends on the data. Edit-distance performed better for ITEM data, while IDF-based retrieval was superior for TEXT data.

Key Results

  • Improved Translation Accuracy: The combined TM+NMT approach significantly improved translation performance on the Timely Disclosure corpus compared to a standalone NMT system.
  • Closing the Gap with Commercial Systems: The hybrid system managed to fill the BLEU score gap between their NMT systems and online black-box MT systems by approximately 1-2 BLEU points.
  • Human Evaluation: Human evaluation showed that the TM+NMT system's translation quality was comparable to human references for the ITEM testset. For the TEXT testset, while the system's performance was lower than human references, it was still competitive.
  • Specific Scores:
    • On the ITEM dataset, the best sacreBLEU scores achieved with TM were around 57.5 (Marian) and 56.6 (Fairseq).
  • On the TEXT dataset, scores reached up to 61.38 (Marian) and 57.7 (Fairseq).
  • Human evaluation mean scores for ITEM were around 71-73, and for TEXT around 66-73, with some systems being indistinguishable from human references.

Limitations and Future Work

  • Longer Sentences and Variability: The paper acknowledges that the TM+NMT approach is less effective for longer, more varied sentences in the TEXT dataset. This is partly because longer sentences with major semantic errors might still be selected due to high similarity scores, and also because commercial systems benefit from much larger training corpora that cover more expressions and domains.
  • Future Work: The authors suggest that improving NMT models for longer sentences, particularly in the context of document-level translation, remains an important area for future research. They also highlight the need to explore more sophisticated ways to integrate TM and NMT, potentially beyond simple threshold-based selection. The effectiveness of the approach was confirmed even with black-box MT systems, suggesting broader applicability.

Summary updated

Abstract

In this paper, we report our submission systems (geoduck) to the Timely Disclosure task on the 6th Workshop on Asian Translation (WAT) (Nakazawa et al., 2019). Our system employs a combined approach of translation memory and Neural Machine Translation (NMT) models, where we can select final translation outputs from either a translation memory or an NMT system, when the similarity score of a test source sentence exceeds the predefined threshold. We observed that this combination approach significantly improves the translation performance on the Timely Disclosure corpus, as compared to a standalone NMT system. We also conducted source-based direct assessment on the final output, and we discuss the comparison between human references and each system’s output.
BibTeX
@inproceedings{eriguchi2019combining,
  author = {Akiko Eriguchi and Spencer Rarrick and Hitokazu Matsushita},
  title = {{Combining Translation Memory with Neural Machine Translation.}},
  booktitle = {WAT @ EMNLP},
  year = {2019},
  url = {https://openreview.net/forum?id=cto640Jjf6g},
}
Figure