IntelliCAT: Intelligent Machine Translation Post-Editing with Quality Estimation and Translation Suggestion

Dongjun Lee, Junhyeong Ahn, Heesoo Park, Jaemin JoACL 2021

TL;DR

This paper introduces IntelliCAT, an interactive machine translation (MT) post-editing tool that significantly speeds up the translation process. It achieves this by integrating advanced quality estimation (QE) and context-aware translation suggestions, leading to a remarkable 52.9% speedup in translation time compared to translating from scratch.

What Problem Does This Paper Address?

Professional translators often use Computer-Assisted Translation (CAT) tools to improve efficiency. Two common approaches are post-editing (PE), where translators correct pre-translated MT output, and interactive translation prediction (ITP), which suggests words or phrases as the translator types. While PE can be faster, it often involves high cognitive load. ITP, on the other hand, can be less efficient and lacks subsegment suggestions. This paper addresses the challenge of combining the efficiency of PE with the helpfulness of ITP, aiming to reduce translator workload and improve translation quality by providing intelligent assistance directly within the CAT interface.

Methods

IntelliCAT is a web-based interface that presents the original and machine-translated documents side-by-side. Its core innovations lie in three key features:

  • Quality Estimation (QE): IntelliCAT employs both sentence-level QE (to predict the overall quality of a sentence) and word-level QE (to pinpoint specific words or phrases needing correction). This helps translators quickly identify areas requiring attention.
  • Translation Suggestion: A novel, context-aware translation suggestion model is introduced. Unlike traditional ITP that only considers the left context, this model leverages both left and right contexts of a selected word or phrase to offer more accurate and relevant suggestions. It's built upon XLM-RoBERTa and fine-tuned with a modified translation language modeling objective.
  • Automatic Formatting: The system uses word alignments to automatically transfer formatting (like bolding, italics, hyperlinks, and footnotes) from the source document to the translated document, saving translators the tedious task of reformatting.

Datasets

  • Machine Translation (MT) Model Training: The English-German parallel corpus from the 2020 News Translation Task was used.
  • Quality Estimation (QE) and Translation Suggestion Training:
    • A large, artificially generated QE dataset was used for initial pre-training.
  • The WMT 2020 English-German QE dataset, consisting of 7,000 source, MT, and post-edited sentence triplets, was used for fine-tuning.
  • Evaluation: The WMT 2020 English-German QE dataset was used for quantitative experiments.

Key Insights and Ideas

  • Context is King for Suggestions: The paper highlights that considering both left and right context for translation suggestions significantly improves their accuracy and usefulness, addressing a key limitation of traditional ITP.
  • QE as a Translator's Compass: Integrating QE directly into the interface acts as a powerful guide, allowing translators to efficiently pinpoint problematic sentences and words, thereby reducing cognitive load.
  • Synergy of Features: The combination of QE, translation suggestions, and automatic formatting creates a synergistic effect, leading to substantial improvements in both translation quality and speed.
  • XLM-RoBERTa's Power: Leveraging a pre-trained cross-lingual language model like XLM-RoBERTa provides a strong foundation for both QE and translation suggestion tasks, enabling state-of-the-art performance.

Key Results

  • Quantitative Improvement: Post-editing with IntelliCAT's proposed QE and suggestion models significantly improved translation quality, achieving a reduction in TER (Translation Error Rate) of up to -6.01 and an increase in BLEU score of +6.15 compared to the baseline MT.
  • Speedup: A user study revealed that IntelliCAT achieved a 52.9% speedup in translation time compared to translating from scratch (using MSWord as a baseline). The "Full" IntelliCAT interface (with all features) was significantly faster than an MT-only interface and MSWord.
  • Usability: The system received an excellent usability score (SUS) of 88.61 ± 7.82, indicating high user satisfaction. Participants found the QE useful for proofreading and translation suggestions convenient for reducing dictionary reliance.

Limitations and Future Work

  • Limitations: The paper does not explicitly detail limitations beyond the inherent challenges of MT and QE. The user study was conducted with a small sample size (nine participants).
  • Future Work: The authors plan to build a pipeline for continuous model improvement using automatically collected source-MT-post-edited triplets. They also aim to implement an automatic post-editing model to further enhance MT performance and explore online learning for QE.

Summary updated

BibTeX
@inproceedings{lee2021intellicat,
  author = {Dongjun Lee and Junhyeong Ahn and Heesoo Park and Jaemin Jo},
  title = {{IntelliCAT: Intelligent Machine Translation Post-Editing with Quality Estimation and Translation Suggestion}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure