Bring More Attention to Syntactic Symmetry for Automatic Postediting of High-Quality Machine Translations

Baikjin Jung, Myungji Lee, Jong-Hyeok Lee, Yunsu KimACL 2023 · DOI

Abstract

Automatic Postediting (APE) systems often struggle to improve machine translations that are already high-quality. This paper introduces a novel regularization technique that injects linguistic "inductive bias" into Transformer models by encouraging them to attend to the symmetrical structure of German sentences. By penalizing skewed self-attention distributions based on the Doppelbaum theory, the researchers achieved significant improvements in translation quality, particularly in perfectly correcting subtle structural errors that standard models often overlook.

Introduction

Automatic Postediting (APE) serves as a critical refinement layer for machine translation (MT), aiming to transform raw MT output into human-parity text. However, as MT systems improve, APE faces a "paradoxical wall": the higher the quality of the initial translation, the more difficult it is for an APE system to provide meaningful improvements without introducing new errors. When an MT output is already 90% correct, the remaining flaws are typically deeply structural or contextual. Standard Transformer models, despite their power, lack an inherent understanding of specific target-language syntactic rules, such as the complex "bracket" structure (Feldermodell) found in German.

The Doppelbaum Approach

To address these structural challenges, the authors propose a linguistically motivated loss function called LDOPPELBAUM. This method is rooted in the Doppelbaum (double tree) theory, which describes the underlying symmetry in German sentence structures where related components (like a verb and its prefix) are often separated by significant distances.

The Doppelbaum (double tree) structure illustrating the syntactic symmetry in German sentences.

Symmetric Self-Attention

Rather than allowing the Transformer's attention mechanism to distribute weights without constraint, the researchers introduced a regularization term to minimize the skewness of the self-attention distribution. By calculating the difference between attention weights on the left versus the right halves of a sentence, the model is encouraged to "look" at the structural brackets (the left and right Satzklammer) more evenly. This "symmetry pressure" is applied dynamically using a learned coefficient ($\alpha$), allowing the model to determine when this linguistic bias is most beneficial during training.

Experimental Setup

The study focused on the English–German language pair within the IT domain, utilizing the following datasets:

  • Training: A combination of the eSCAPE-NMT synthetic corpus (approx. 5 million segments) and the WMT 2019 training set.
  • Evaluation: The WMT 2019 test set, which is characterized by high-quality translations that are notoriously difficult for APE systems to enhance.

Results and Discussion

The proposed DOPPELBAUM model demonstrated superior performance over the baseline across multiple metrics. Notably, it achieved a BLEU score of 75.47, a statistically significant improvement over the baseline (75.11) and the original MT (74.73).

Performance breakdown of the Baseline vs. DOPPELBAUM models, highlighting improvements in perfect corrections (PERF) and F1 scores.

One of the most significant findings was the model's ability to handle "nontrivial" edits. Manual analysis revealed that the model was particularly effective at fixing linguistic errors—such as noun-verb agreement and preposition usage—where related words were far apart. As shown in the performance breakdown, the DOPPELBAUM model increased the proportion of "Perfect" corrections (PERF) to 5.87%, compared to the baseline's 4.30%. Furthermore, the system achieved an F1 score of 25.4 (vs. 22.8 for the baseline), indicating a better ability to decide exactly when an edit is necessary and how to execute it correctly.

Conclusion

This research highlights that for high-quality MT, simply increasing data volume is not always the solution; instead, providing the model with a better "mental model" of the language through inductive bias is crucial. By forcing the attention mechanism to respect linguistic symmetry, the DOPPELBAUM model successfully identifies long-distance dependencies that standard Transformers miss. While the current implementation is specific to German syntax, the concept of structural attention constraints offers a promising path for improving APE in other languages with unique syntactic symmetries, such as Japanese or Korean.

Summary updated

BibTeX
@inproceedings{jung2023bring,
  author = {Baikjin Jung and Myungji Lee and Jong-Hyeok Lee and Yunsu Kim},
  title = {{Bring More Attention to Syntactic Symmetry for Automatic Postediting of High-Quality Machine Translations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-short.122},
}
Figure