A Post-editing Interface for Immediate Adaptation in Statistical Machine Translation

Patrick Simianer, Sariya Karimova, Stefan RiezlerCOLING 2016

TL;DR

This paper introduces an open-source post-editing interface for adaptive statistical machine translation (SMT) that allows for immediate learning from user corrections. By providing a rich graphical interface that visualizes alignments and enabling novel adaptation techniques, the system demonstrably reduces technical effort and time for translators.

What Problem Does This Paper Address?

While adaptive machine translation (MT) systems hold great promise for improving the efficiency of computer-aided translation (CAT) environments, there's been a significant gap between theoretical proposals and practical implementations. Existing research often simulates user behavior or treats MT engines as static black boxes. This paper addresses the need for a practical, open-source toolkit that facilitates real user studies in adaptive MT environments, enabling the investigation of how users interact with and how MT systems learn from these interactions to improve translation quality and translator productivity.

Methods

The core contribution is a novel post-editing interface designed for adaptive SMT. This interface offers both a standard text-based view and a richer graphical view that displays latent segmentations and alignments between source and target phrases. This graphical representation allows users to not only correct translations but also to refine the alignments, providing more granular feedback than simple string edits.

The adaptation methods build upon existing techniques but leverage the richer feedback from the graphical interface. Key aspects include:

  • Out-of-Vocabulary (OOV) Handling: An immediate feedback mechanism prompts users for translations of unknown words before decoding, preventing translation failures.
  • Grammar Adaptation: User-corrected alignments are used to immediately add lexical corrections and new translation rules to the current grammar.
  • Weight Adaptation: The system re-decodes the source with the augmented grammar, generates a k-best list, and re-ranks it using BLEU+1 with the post-edit as a reference. Weights are updated based on this list.
  • Novel Rule Extraction: A rule extraction process, similar to Chiang's (2007) grammar extraction, is adapted to use full phrases from the alignment, generating a large number of additional rules. To prevent overfitting, these rules are added incrementally.
  • Adaptive Language Model: The language model is updated with the string of the current post-edit.

Datasets

The paper does not explicitly name specific large-scale datasets used for training the initial MT models. However, for the user study, patent titles and abstracts were used. These documents were limited to a maximum length of 45 tokens per sentence and were grouped into clusters by cosine similarity of their bag-of-words tf-idf representations and length, creating clusters of approximately 500 source tokens. This approach aimed to provide related documents for a coherent post-editing task.

Key Insights and Ideas

  • Rich Alignment Feedback is Crucial: Visualizing and allowing users to edit phrase alignments, not just target strings, provides significantly more valuable information for adapting MT systems compared to surface-level corrections alone.
  • Immediate Adaptation Matters: The ability to adapt the MT system in near real-time during a post-editing session can lead to tangible improvements in translator efficiency.
  • Bridging the Gap Between Theory and Practice: The development of an open-source, feature-rich toolkit is essential for enabling rigorous user studies and advancing research in adaptive MT.
  • User Expectations Drive Tool Development: Translators are increasingly expecting adaptive tools, similar to how translation memories work, making this research highly relevant to industry needs.

Key Results

A user study with 19 participants using the proposed adaptive MT system showed:

  • Significant improvements in translation quality: Measured by HBLEU+1 (increase of +6.8 ± 2.0 [%], p < 0.001).
  • Significant reduction in technical effort: Measured by HTER (decrease of -5.3 ± 1.9 [%], p < 0.01).
  • Non-significant time reduction: A slight reduction in normalized time (-118 ms) was observed but did not reach statistical significance.
  • Overall post-edit quality, measured by BLEU+1 scores against reference translations, remained stable at approximately 39.6 ± 0.4 [%].

Limitations and Future Work

  • Non-significant time reduction: While quality and effort metrics improved, the time savings were not statistically significant. This could be due to the complexity of the task, the learning curve of the interface, or the specific adaptation methods employed.
  • Focus on SMT: The current implementation is based on a hierarchical phrase-based SMT engine. Future work could extend these adaptation techniques to neural MT systems.
  • Further Exploration of Adaptation Techniques: The paper suggests that the rich feedback from the graphical interface could enable even more novel and precise adaptation methods.
  • Broader User Studies: While the study involved diverse participants, further studies with professional translators in real-world scenarios would be valuable.
  • Source Code Availability: The authors have released the source code under the LGPL, which is a significant step for enabling further research and development in this area.

Summary updated

BibTeX
@inproceedings{simianer2016post,
  author = {Patrick Simianer and Sariya Karimova and Stefan Riezler},
  title = {{A Post-editing Interface for Immediate Adaptation in Statistical Machine Translation}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2016},
}
Figure