TL;DR
While machine translation systems can be forced to use specific domain terminologies, downstream Automatic Post-Editing (APE) systems often inadvertently overwrite or drop these constraints. This paper introduces lexically constrained APE models using both autoregressive and non-autoregressive architectures, successfully preserving over 95% of target terms while simultaneously improving translation quality.
What Problem Does This Paper Address?
Automatic Post-Editing (APE) acts as a corrector for machine translation (MT), fixing systematic errors and adapting text to specific domains. However, real-world translation often requires strict adherence to specific terminology constraints—such as brand names, technical jargon, or user queries in cross-lingual search.
While modern MT systems can handle these constraints at inference time, standard APE systems are completely blind to them. As a result, when you pass a perfectly constrained MT output through a vanilla APE system, it often "corrects" the specialized terminology back into generic text. The authors show that applying an unconstrained APE system to a constrained MT output causes a 12.6% relative drop in preserved terminology. This paper tackles the challenge of making APE systems constraint-aware without sacrificing their ability to fix translation errors.
Methods
The authors propose combining input sequence modification strategies with two distinct neural architectures to build a constraint-aware APE pipeline:
- Encoding Constraints in the Input: To tell the model which words are constraints, they modify the source text sequence ($x$) using two methods:
- Append ($x^+$): The target language constraint is inserted immediately after its corresponding source word.
- Replace ($x^-$): The source word is completely replaced by the target language constraint.
To help the model differentiate between normal source words, source constraints, and target constraints, they use "source factors"—additional embedding layers that tag each token with a feature ID (0 to 3).
- Model Architectures:
- Multi-source Transformer (MST): An autoregressive model that uses two separate encoders—one for the source text (plus constraints) and one for the initial MT output ($\gamma$)—which are concatenated and fed into a standard decoder.
- Levenshtein Transformer (LevT): A non-autoregressive model that learns to edit text via deletion and insertion operations. The authors introduce a variant called MS LevT, where the decoder is directly initialized with the target terminology constraints, bypassing the typical blank-slate initialization.
- Data Augmentation for Robustness: The authors discovered that models often "cheat" by relying on the language model's familiarity with common terms rather than actually learning to copy constraints. To force true copying behavior, they injected synthetic training data where target constraints were replaced with completely unexpected synonyms or antonyms.
Datasets
The models were evaluated on standard English-to-German APE benchmarks from the IT domain:
- WMT18 PBMT: Based on phrase-based machine translation outputs.
- WMT19 NMT: Based on neural machine translation outputs.
- Synthetic Pretraining Corpora: The models were pretrained on large synthetic datasets (artificial 4M and eSCAPE NMT) before fine-tuning on official data.
- Terminology Extraction: Constraint pairs were automatically harvested using Wiktionary, matching source words to their exact post-edited targets while filtering out stop words.
Key Insights and Ideas
- The "Constraint Erasure" Dilemma: Standard APE acts as a bottleneck for constrained translation. Even if your MT system perfectly respects a glossary, a standard downstream APE will degrade that performance because its internal language model prefers high-frequency generic words.
- Models Don't Naturally Copy: Simply feeding constraints into a model doesn't mean it learns a robust "copying" mechanism. When tested with unusual or semantically divergent constraints (like antonyms), standard constrained models frequently ignore them.
- The Power of Semantic Divergence: Training models on intentionally "wrong" constraints (via data augmentation with synonyms/antonyms) acts as a regularizer. It breaks the model's reliance on language-model probability and forces it to treat the constraint token as an absolute command to copy.
Key Results
The constrained APE approaches consistently outperformed the "do-nothing" baseline and unconstrained models in terminology preservation, achieving over 95% coverage.
- On PBMT 2018: The
MS LevTmodel achieved the highest terminology preservation rate at 97.50%, though theMST Appendmodel provided a better balance of translation quality (TER of 18.97 vs 20.39). - On NMT 2019:
MS LevTreached an impressive 98.04% terminology coverage. Meanwhile,MST Appendsuccessfully boosted terminology preservation to 94.08% (up from the baseline's 90.22%) while maintaining competitive BLEU and TER scores.
Limitations and Future Work
The authors note a clear trade-off between constraint sensitivity and output stability. As models become better at forcing constraints into the text (higher Term%), their Translation Error Rate (TER) tends to increase. This happens because the model makes sweeping structural changes to the surrounding sentence to accommodate an unusual word, rather than making a surgical, minimal edit.
Future work will focus on developing explicit loss functions or architectural constraints that encourage output stability, ensuring that forcing a terminology constraint doesn't disrupt the rest of an otherwise correct sentence.
Summary updated