Abstract
Automatic Postediting (APE) systems often struggle to improve machine translations that are already high-quality. This paper introduces a novel regularization technique that injects linguistic "inductive bias" into Transformer models by encouraging them to attend to the symmetrical structure of German sentences. By penalizing skewed self-attention distributions based on the Doppelbaum theory, the researchers achieved significant improvements in translation quality, particularly in perfectly correcting subtle structural errors that standard models often overlook.
Introduction
Automatic Postediting (APE) serves as a critical refinement layer for machine translation (MT), aiming to transform raw MT output into human-parity text. However, as MT systems improve, APE faces a "paradoxical wall": the higher the quality of the initial translation, the more difficult it is for an APE system to provide meaningful improvements without introducing new errors. When an MT output is already 90% correct, the remaining flaws are typically deeply structural or contextual. Standard Transformer models, despite their power, lack an inherent understanding of specific target-language syntactic rules, such as the complex "bracket" structure (Feldermodell) found in German.
The Doppelbaum Approach
To address these structural challenges, the authors propose a linguistically motivated loss function called LDOPPELBAUM. This method is rooted in the Doppelbaum (double tree) theory, which describes the underlying symmetry in German sentence structures where related components (like a verb and its prefix) are often separated by significant distances.
Symmetric Self-Attention
Rather than allowing the Transformer's attention mechanism to distribute weights without constraint, the researchers introduced a regularization term to minimize the skewness of the self-attention distribution. By calculating the difference between attention weights on the left versus the right halves of a sentence, the model is encouraged to "look" at the structural brackets (the left and right Satzklammer) more evenly. This "symmetry pressure" is applied dynamically using a learned coefficient ($\alpha$), allowing the model to determine when this linguistic bias is most beneficial during training.
Experimental Setup
The study focused on the English–German language pair within the IT domain, utilizing the following datasets:
- Training: A combination of the eSCAPE-NMT synthetic corpus (approx. 5 million segments) and the WMT 2019 training set.
- Evaluation: The WMT 2019 test set, which is characterized by high-quality translations that are notoriously difficult for APE systems to enhance.
Results and Discussion
The proposed DOPPELBAUM model demonstrated superior performance over the baseline across multiple metrics. Notably, it achieved a BLEU score of 75.47, a statistically significant improvement over the baseline (75.11) and the original MT (74.73).
One of the most significant findings was the model's ability to handle "nontrivial" edits. Manual analysis revealed that the model was particularly effective at fixing linguistic errors—such as noun-verb agreement and preposition usage—where related words were far apart. As shown in the performance breakdown, the DOPPELBAUM model increased the proportion of "Perfect" corrections (PERF) to 5.87%, compared to the baseline's 4.30%. Furthermore, the system achieved an F1 score of 25.4 (vs. 22.8 for the baseline), indicating a better ability to decide exactly when an edit is necessary and how to execute it correctly.
Conclusion
This research highlights that for high-quality MT, simply increasing data volume is not always the solution; instead, providing the model with a better "mental model" of the language through inductive bias is crucial. By forcing the attention mechanism to respect linguistic symmetry, the DOPPELBAUM model successfully identifies long-distance dependencies that standard Transformers miss. While the current implementation is specific to German syntax, the concept of structural attention constraints offers a promising path for improving APE in other languages with unique syntactic symmetries, such as Japanese or Korean.
Summary updated