Introduction
Machine translation (MT) often suffers from a "consistency problem." When an MT engine is trained on data that does not perfectly match a specific document's domain, it frequently mistranslates technical terms or phrases identically every time they appear. For human post-editors, this creates a repetitive and frustrating workflow—correcting the same error dozens of times within a single manual, a phenomenon the authors describe as "document-internal repetition."
While Translation Memory (TM) systems can suggest previous translations for identical sentences, they lack the ability to "learn" and fix systematic errors within new, similar sentences. PEPr (Post-Edit Propagation) addresses this technological gap by observing human corrections in real-time and proactively applying those fixes to subsequent sentences within the same document.
Methodology
PEPr utilizes a phrase-based Statistical Machine Translation (SMT) system in an Automatic Post-Editing (APE) configuration. Rather than attempting to modify the core MT engine, PEPr operates as a secondary layer that treats the raw MT output as a "source language" and the human's corrected version as the "target language."
Key technical components of the framework include:
- Incremental Learning: As a translator completes each sentence, the pair of (MT output, Human correction) is immediately used to update the system's phrase table and language models.
- The "Do-Nothing" Option: To prevent over-correction or "hallucinations," the system includes "identity" phrase pairs in its training. This allows PEPr to pass MT output through unchanged when confidence in a specific fix is low.
- Sparse Data Alignment: Because PEPr starts with zero data for each new document, standard alignment tools are insufficient. The authors employ Damerau-Levenshtein distance to align words between MT output and human corrections.
- Language Model (LM) Mixing: The system balances a "Reference LM" (built from recent human corrections) with either an "MT LM" (from the raw document) or a "Generic LM" (background data) to maintain fluency.
Experimental Setup
The researchers evaluated PEPr across four datasets representing varying levels of repetitivity:
- ECB: European Central Bank documents (High repetition).
- EMEA: European Medicines Agency documents (High repetition).
- Science Abstracts: Short, highly varied summaries (Low repetition).
- Science Digests: Grouped abstracts simulating longer, coherent documents.
To simulate a realistic environment, documents were truncated to 100 sentences, and pre-existing human translations served as the "gold standard" for post-edits.
Results and Discussion
The effectiveness of PEPr is highly dependent on the repetitivity of the source text. In high-repetition domains like legal and medical documentation (ECB and EMEA), the system demonstrated substantial performance gains.
Key findings include:
- Significant Metric Gains: In the ECB and EMEA datasets, PEPr boosted BLEU scores by 5 to 9 points and reduced the Word Error Rate (WER) by 3 to 6 points.
- The Value of Generic Data: Mixing in a Generic Language Model (+GLM) consistently outperformed using local data alone, providing a necessary "sanity check" for fluency that small, document-specific datasets cannot offer.
- Limits of Low Repetition: On the Science Abstracts dataset, the system provided negligible or even slightly negative results. Without sufficient patterns to learn from, the propagation mechanism cannot effectively predict corrections.
Conclusion
PEPr demonstrates that focusing on "local" knowledge within a single document can be more effective for post-editing than relying solely on global training. By treating the MT engine as a black box, PEPr remains compatible with any underlying technology, including modern neural models. While it is a specialized tool best suited for technical and legal prose rather than creative text, it offers a promising path toward reducing the manual labor involved in professional translation workflows. Future work will focus on testing the system with real human translators to measure actual time savings and refining phrase-table filtering to improve reliability.
Summary updated