Incorporating Terminology Constraints in Automatic Post-Editing

David Wan, Chris Kedzie, Faisal Ladhak, Marine Carpuat, Kathleen McKeown

TL;DR

While machine translation systems can be forced to use specific domain terminologies, downstream Automatic Post-Editing (APE) systems often inadvertently overwrite or drop these constraints. This paper introduces lexically constrained APE models using both autoregressive and non-autoregressive architectures, successfully preserving over 95% of target terms while simultaneously improving translation quality.

What Problem Does This Paper Address?

Automatic Post-Editing (APE) acts as a corrector for machine translation (MT), fixing systematic errors and adapting text to specific domains. However, real-world translation often requires strict adherence to specific terminology constraints—such as brand names, technical jargon, or user queries in cross-lingual search.

While modern MT systems can handle these constraints at inference time, standard APE systems are completely blind to them. As a result, when you pass a perfectly constrained MT output through a vanilla APE system, it often "corrects" the specialized terminology back into generic text. The authors show that applying an unconstrained APE system to a constrained MT output causes a 12.6% relative drop in preserved terminology. This paper tackles the challenge of making APE systems constraint-aware without sacrificing their ability to fix translation errors.

Methods

The authors propose combining input sequence modification strategies with two distinct neural architectures to build a constraint-aware APE pipeline:

  1. Encoding Constraints in the Input: To tell the model which words are constraints, they modify the source text sequence ($x$) using two methods:
  2. Append ($x^+$): The target language constraint is inserted immediately after its corresponding source word.
  3. Replace ($x^-$): The source word is completely replaced by the target language constraint.

To help the model differentiate between normal source words, source constraints, and target constraints, they use "source factors"—additional embedding layers that tag each token with a feature ID (0 to 3).

Example of Append and Replace methods with source factors
  1. Model Architectures:
  2. Multi-source Transformer (MST): An autoregressive model that uses two separate encoders—one for the source text (plus constraints) and one for the initial MT output ($\gamma$)—which are concatenated and fed into a standard decoder.
  3. Levenshtein Transformer (LevT): A non-autoregressive model that learns to edit text via deletion and insertion operations. The authors introduce a variant called MS LevT, where the decoder is directly initialized with the target terminology constraints, bypassing the typical blank-slate initialization.
Inference setup for the various AT and NAT model configurations
  1. Data Augmentation for Robustness: The authors discovered that models often "cheat" by relying on the language model's familiarity with common terms rather than actually learning to copy constraints. To force true copying behavior, they injected synthetic training data where target constraints were replaced with completely unexpected synonyms or antonyms.

Datasets

The models were evaluated on standard English-to-German APE benchmarks from the IT domain:

  • WMT18 PBMT: Based on phrase-based machine translation outputs.
  • WMT19 NMT: Based on neural machine translation outputs.
  • Synthetic Pretraining Corpora: The models were pretrained on large synthetic datasets (artificial 4M and eSCAPE NMT) before fine-tuning on official data.
  • Terminology Extraction: Constraint pairs were automatically harvested using Wiktionary, matching source words to their exact post-edited targets while filtering out stop words.

Key Insights and Ideas

  • The "Constraint Erasure" Dilemma: Standard APE acts as a bottleneck for constrained translation. Even if your MT system perfectly respects a glossary, a standard downstream APE will degrade that performance because its internal language model prefers high-frequency generic words.
  • Models Don't Naturally Copy: Simply feeding constraints into a model doesn't mean it learns a robust "copying" mechanism. When tested with unusual or semantically divergent constraints (like antonyms), standard constrained models frequently ignore them.
  • The Power of Semantic Divergence: Training models on intentionally "wrong" constraints (via data augmentation with synonyms/antonyms) acts as a regularizer. It breaks the model's reliance on language-model probability and forces it to treat the constraint token as an absolute command to copy.

Key Results

The constrained APE approaches consistently outperformed the "do-nothing" baseline and unconstrained models in terminology preservation, achieving over 95% coverage.

  • On PBMT 2018: The MS LevT model achieved the highest terminology preservation rate at 97.50%, though the MST Append model provided a better balance of translation quality (TER of 18.97 vs 20.39).
  • On NMT 2019: MS LevT reached an impressive 98.04% terminology coverage. Meanwhile, MST Append successfully boosted terminology preservation to 94.08% (up from the baseline's 90.22%) while maintaining competitive BLEU and TER scores.
Quantitative results on the WMT19 NMT dataset
Table 5: Results for PBMT 2018.

Limitations and Future Work

The authors note a clear trade-off between constraint sensitivity and output stability. As models become better at forcing constraints into the text (higher Term%), their Translation Error Rate (TER) tends to increase. This happens because the model makes sweeping structural changes to the surrounding sentence to accommodate an unusual word, rather than making a surgical, minimal edit.

Future work will focus on developing explicit loss functions or architectural constraints that encourage output stability, ensuring that forcing a terminology constraint doesn't disrupt the rest of an otherwise correct sentence.

Summary updated

Abstract

Users of machine translation (MT) may want to ensure the use of specific lexical terminologies. While there exist techniques for incorporating terminology constraints during inference for MT, current APE approaches cannot ensure that they will appear in the final translation. In this paper, we present both autoregressive and non-autoregressive models for lexically constrained APE, demonstrating that our approach enables preservation of 95% of the terminologies and also improves translation quality on English-German benchmarks. Even when applied to lexically constrained MT output, our approach is able to improve preservation of the terminologies. However, we show that our models do not learn to copy constraints systematically and suggest a simple data augmentation technique that leads to improved performance and robustness.
BibTeX
@inproceedings{wan2020incorporating,
  author = {David Wan and Chris Kedzie and Faisal Ladhak and Marine Carpuat and Kathleen McKeown},
  title = {{Incorporating Terminology Constraints in Automatic Post-Editing}},
  booktitle = {WMT @ EMNLP},
  year = {2020},
  url = {https://openreview.net/forum?id=iZQ3RlO7D1j},
}
Figure