Contextual Refinement of Translations: Large Language Models for Sentence and Document-Level Post-Editing

Sai Koneru, Miriam Exel, Matthias Huck, Jan Niehues

Introduction

While Large Language Models (LLMs) have demonstrated remarkable capabilities across various natural language tasks, they often struggle to surpass specialized Neural Machine Translation (NMT) systems in raw translation quality. This paper explores a novel paradigm: instead of using LLMs as primary translators, they are employed as Automatic Post-Editors (APE) to refine the output of existing NMT models. By leveraging parameter-efficient fine-tuning and document-level context, the authors achieve significant improvements in translation accuracy and contextual consistency.

The Challenge of Document-Level Translation

Standard NMT systems typically operate on a sentence-by-sentence basis. This localized focus often leads to "document-level" errors, such as inconsistent pronoun usage or mismatched tone, because the model lacks the broader context of the surrounding text. Furthermore, direct fine-tuning of LLMs for translation tasks can sometimes degrade performance compared to simple zero-shot prompting—a phenomenon dubbed the "translation paradox."

Proposed Methodology: A Cascaded Approach

The researchers propose a modular system that combines the precision of NMT with the contextual awareness of LLMs. The workflow follows a cascaded structure:

  1. Initial Drafting: A dedicated NMT model (DeltaLM) generates a preliminary translation.
  2. LLM Refinement: A Llama-2-13b-chat model takes both the source sentence and the NMT draft as input to produce a polished version.
  3. Contextual Windowing: To handle document-level dependencies, the system uses sliding windows to provide the LLM with preceding sentences, ensuring consistency across the text.
  4. Iterative Human-in-the-Loop: The system supports a workflow where manual corrections are fed back into the LLM's prompt as "gold context," improving the translation of subsequent sentences.
Overview of the proposed cascaded translation and post-editing workflow.

To optimize the LLM for this task, the authors utilized Quantized Low-Rank Adaptation (Q-LoRA). They simulated NMT errors during training to teach the LLM specifically how to identify and rectify common machine translation artifacts.

Experimental Setup and Datasets

The approach was evaluated using several key benchmarks, including MuST-C V3 (English-to-German TED talk transcripts) and out-of-domain datasets like WMT21 News and ACL Dev. Additionally, the ContraPro benchmark was used to specifically test pronoun resolution, a common failure point for sentence-level systems.

Results and Discussion

The experimental results demonstrate that LLMs are more effective as "editors" than "authors" in the translation domain. By providing an NMT hypothesis, the search space for the LLM is reduced, allowing it to focus on fluency and long-range consistency.

Performance comparison across different datasets showing the superiority of the combined APE approach.

Key findings include:

  • State-of-the-Art Pronoun Resolution: The model achieved 88.7% accuracy on the ContraPro test set, outperforming previous specialized document-level NMT systems.
  • Robust Generalization: The APE approach showed significant gains on out-of-domain data, improving BLEU scores by up to 5.64 points on the ACL dev set compared to the base NMT model.
  • Synergy with Human Feedback: Incorporating human-corrected "Gold Target Context" resulted in a substantial performance boost (+4.14 BLEU), creating a virtuous cycle where manual edits improve subsequent automated refinements.

Conclusion

This research highlights the potential of LLMs as sophisticated post-editing tools. By framing translation as a refinement process rather than a generation task, the authors successfully bypassed the limitations of direct LLM translation. While computational latency remains a challenge for real-time applications, the significant gains in document-level consistency and pronoun resolution mark a major step forward in high-quality machine translation.

Summary updated

BibTeX
@inproceedings{koneru2024contextual,
  author = {Sai Koneru and Miriam Exel and Matthias Huck and Jan Niehues},
  title = {{Contextual Refinement of Translations: Large Language Models for Sentence and Document-Level Post-Editing}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.148},
}
Figure