TL;DR
This paper investigates how Large Language Models (LLMs) leverage different types of context (previous source sentences, model's own hypotheses, or gold references) to improve discourse-aware machine translation. They found that context consistently boosts translation quality and specific discourse phenomena, with a promising strategy of using the model's own prior hypotheses to achieve discourse consistency without relying on perfect reference translations.
What Problem Does This Paper Address?
While Large Language Models (LLMs) have revolutionized machine translation (MT), achieving human-quality translations, they still struggle with "discourse-level phenomena." These are the nuances that make a translation sound natural and coherent across multiple sentences, such as maintaining consistent formality (e.g., polite vs. casual speech) or ensuring pronouns refer to the correct entities. Existing research often focuses on overall translation quality (e.g., BLEU scores) rather than how LLMs specifically use context to tackle these finer points. This paper aims to bridge that gap by systematically exploring how different contextual strategies impact discourse-aware MT.
Methods
The authors designed experiments to test various context strategies for LLM-based MT. They defined several ways to provide context ($C_n$) to the LLM when translating a source sentence ($s_n$) to generate a hypothesis ($h_n$):
- Source Only (src): Using the $k$ preceding source sentences.
- Hypothesis Only (hyp): Using the $k$ previously generated hypotheses by the LLM.
- Hypothesis with Source (src+hyp): Combining both preceding source sentences and their corresponding generated hypotheses. This is a key strategy as it uses the LLM's own (imperfect) output as context.
- Reference Only (ref): Using the $k$ preceding gold-standard reference translations (for evaluation, not real-world use).
- Reference with Source (src+ref): Combining preceding source sentences and their reference translations.
They evaluated these strategies on two leading LLMs (GPT-40 and Gemma 2-27B) across multiple language pairs (English-to-Japanese, English-to-French, English-to-German) and two distinct domains (TED Talks monologues and OpenSubtitles dialogues). To measure discourse quality, they employed the MuDA framework, which specifically assesses formality, lexical cohesion, and pronoun selection, going beyond standard MT metrics like BLEU and COMET.
Datasets
The study utilized two main datasets:
- TED Talks Corpus: This corpus, specifically the IWSLT section, was used for multilingual monologues. Test sets were created for English-to-Japanese (5565 sentences), English-to-French (4866 sentences), and English-to-German (4491 sentences).
- OpenSubtitles Subset: A manually curated subset of the OpenSubtitles2018 corpus was created for English-to-Japanese conversational dialogue. This subset contained 3961 dialogue turns, carefully selected and cleaned to ensure quality and preserve contextual dependencies.
Key Insights and Ideas
- Context is King (for Discourse): All tested context strategies, even just providing the source text, consistently improved both overall translation quality and specific discourse phenomena compared to a no-context baseline.
- Self-Supervision Powerhouse: The
src+hypstrategy, which uses the LLM's own previous hypotheses as context, proved remarkably effective for improving discourse consistency. This is crucial because it works without needing perfect "gold" reference translations, suggesting LLMs can learn from and correct their own outputs. - Diminishing Returns: While more context is generally better, there's an optimal context window size ($k$). Beyond a certain point (around $k=7$ in their experiments), adding more context yields diminishing returns or can even slightly degrade performance, likely due to noise or irrelevant information.
- Formality and Pronoun Fidelity: The
src+hypstrategy showed notable improvements in maintaining formality (e.g., switching from overly formal to informal Japanese) and correctly selecting pronouns (e.g., gender agreement in German), demonstrating its practical utility.
Key Results
- Consistent Improvement: Across all language pairs and datasets, context strategies generally outperformed the no-context baseline. For example, on the TED en-ja task, GPT-40's MuDA F1 score for formality increased from 0.52 (no-ctx) to 0.55 (src+hyp) and even 0.61 (src+ref).
src+hypNearing Gold Standard: Thesrc+hypstrategy often achieved performance close to, and sometimes even surpassing, therefstrategy (which uses perfect reference translations) for discourse metrics, especially for models like Gemma 2-27B.- Correlation of Context Quality: For the
src+hypstrategy, they found a consistent positive correlation between the discourse quality of the context batch and the translated sentence's quality. This reinforces the idea that better context leads to better discourse-aware translations.
Limitations and Future Work
The authors acknowledge several limitations:
- Language and Domain Diversity: While they tested multiple languages and domains, further investigation into more typologically diverse languages and larger-scale datasets is warranted. The OpenSubtitles dataset curation was primarily focused on English-to-Japanese due to expertise requirements.
- Model Opacity: The closed-source nature of models like GPT-40 limits a full understanding of their internal workings and makes direct comparisons with open-source models challenging.
- Evaluation Nuances: The MuDA framework's reliance on surface-form matching might underestimate performance by penalizing valid alternative phrasing.
- Prompting Sensitivity: The study used a single, fixed prompting strategy. Different prompting approaches could potentially yield different results, especially for phenomena like formality.
Future work could involve expanding the analysis to more languages and domains, exploring different prompting techniques, and investigating the impact of context on a wider range of discourse phenomena.
Summary updated