You Are What You Train: Effects of Data Composition on Training Context-aware Machine Translation Models

Paweł Mąka, Yusuf Can Semerci, Jan Scholtes, Gerasimos SpanakisEMNLP 2025

TL;DR

This paper investigates how the proportion of contextually rich examples in training data impacts context-aware machine translation (MT) models. They find that data sparsity is a major bottleneck, and improvements in one contextual phenomenon don't generalize to others. The authors propose and evaluate two methods to better leverage existing data for improved context utilization.

What Problem Does This Paper Address?

Modern machine translation (MT) systems aim to produce human-level translations by leveraging surrounding sentences (context) to ensure coherence and resolve ambiguities. However, a persistent challenge is that these context-aware MT models often struggle to effectively utilize this context. The prevailing hypothesis is that the scarcity of "contextually rich" examples – sentences where context is crucial for accurate translation – in standard training datasets is the root cause of this difficulty. This paper aims to systematically validate this hypothesis and explore ways to overcome it.

Methods

The authors employ a systematic approach to investigate the impact of data composition:

  • Controlled Data Augmentation: They construct training datasets by varying the proportion of contextually rich examples. This involves identifying sentences requiring context for phenomena like gender, formality, auxiliary verbs, and inflection using the ctxPro toolset.
  • Model Training: They train both standard encoder-decoder Transformer models (like OpusMT and NLLB-200) and decoder-only Large Language Models (LLMs) (like Towerbase 7B).
  • Evaluation Metrics: Beyond standard metrics like BLEU and COMET, they heavily rely on ctxPro accuracy to specifically measure performance on context-dependent phenomena.
  • Proposed Training Strategies:
    • Token-level Loss Weighting: This method assigns higher importance to tokens that are contextually dependent during training.
  • Metric-based Example Selection: This approach uses a model's own output probabilities (specifically, a modified Point-wise Cross-Mutual Information metric called MaxPCXMI) to identify and prioritize contextually rich examples for fine-tuning.

Datasets

  • IWSLT 2017 English-to-German: Used for single-language pair experiments.
  • OpenSubtitles 2018: Used for multilingual experiments and as a source for contextually rich examples.
  • ctxPro Dataset: Used to extract contextually rich examples for specific phenomena (Gender, Formality, Auxiliary, Inflection, Animacy) and for evaluation. The paper also describes the construction of "sparse" (low contextually rich examples) and "dense" (high contextually rich examples) versions of these datasets.

Key Insights and Ideas

  • Sparsity is the Bottleneck: The paper strongly confirms that the density of contextually rich examples in training data is directly correlated with a model's ability to leverage context. Increasing this density leads to significant performance gains.
  • Lack of Generalization Across Phenomena: A surprising finding is that improvements in translating one type of contextual phenomenon (e.g., gender) do not transfer to other phenomena (e.g., formality). This suggests models might develop specialized mechanisms for each.
  • Limited Cross-Lingual Transfer: While some cross-lingual transfer of context utilization is observed, it's not significantly stronger between languages within the same linguistic sub-family, contrary to some expectations.
  • LLMs as Reflections of Training Data: The study notes that LLMs, despite their scale, do not inherently exhibit better generalization across phenomena compared to smaller models, reinforcing the idea that their capabilities are largely dictated by their training data.
  • Trade-off Between Context and General Translation: The proposed methods for improving context utilization often come with a slight decrease in general translation metrics like BLEU, highlighting a trade-off that needs careful consideration.

Key Results

  • Accuracy Gains: The proposed training strategies (Token-level Loss Weighting and Metric-based Example Selection) achieved accuracy gains of up to 6 and 8 percentage points on ctxPro evaluations in single- and multilingual settings, respectively.
  • Formality and Auxiliary Improvements: Metric-based example selection, particularly using MaxPCXMI, showed the highest improvements for Formality and Auxiliary phenomena.
  • BLEU vs. ctxPro Accuracy: A consistent observation across methods is a negative correlation between improvements in ctxPro accuracy and BLEU scores, indicating a trade-off.
  • Multilingual Performance: For multilingual models (NLLB-200), MaxPCXMI fine-tuning outperformed baselines, showing an average improvement of 5.8, 1.9, and 1.6 percentage points for Auxiliary, Gender, and Animacy, respectively.

Limitations and Future Work

  • Limited Language Coverage: The study was constrained by the language directions available in the ctxPro toolset, primarily focusing on Indo-European languages.
  • Single Encoder Focus: For encoder-decoder architectures, only the single-encoder approach was tested; multi-encoder models were not explored.
  • Single LLM Model: For decoder-only LLMs, experiments were conducted on a single model (Towerbase 7B), and the behavior might differ across model sizes and families.
  • Future Work: The authors suggest combining the strengths of both weighting and metric-based example selection methods. They also note that assessing complex interconnections between phenomena is left for future work.

Summary updated

BibTeX
@inproceedings{maka2025you,
  author = {Paweł Mąka and Yusuf Can Semerci and Jan Scholtes and Gerasimos Spanakis},
  title = {{You Are What You Train: Effects of Data Composition on Training Context-aware Machine Translation Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure