Contrastive Conditioning for Assessing Disambiguation in MT: A Case Study of Distilled Bias

Jannis Vamvas, Rico SennrichEMNLP 2021 · DOI

Abstract

Machine translation (MT) models frequently struggle with lexical disambiguation, often defaulting to majority senses or gendered stereotypes. While knowledge distillation (SeqKD) is widely used to improve model efficiency, its impact on linguistic diversity remains under-explored. This paper introduces "contrastive conditioning," a reference-free, black-box evaluation method to audit disambiguation performance. Our findings reveal that distillation significantly amplifies overgeneralization bias, leading student models to ignore rare word senses and minority gender forms more frequently than their teacher models or models trained from scratch.

Introduction

The pursuit of efficient machine translation has led to the widespread adoption of sequence-level knowledge distillation (SeqKD). In this paradigm, a compact "student" model is trained to replicate the output distribution of a larger "teacher" model. While this improves inference speed and often maintains high BLEU scores, it may come at a hidden cost: the loss of linguistic nuance. Specifically, distillation may act as a low-pass filter that smooths out the "long tail" of language, reinforcing the most frequent patterns at the expense of rare but correct disambiguations.

Methodology: Contrastive Conditioning

Evaluating disambiguation is challenging, particularly for black-box models where internal probability scores are unavailable. Vamvas and Sennrich propose "contrastive conditioning," a method that flips the traditional evaluation approach. Instead of comparing model probabilities for different translations, the method evaluates a single translation against two versions of the source sentence.

  1. Translation: The target model generates a translation for an ambiguous source sentence.
  2. Source Modification: Two variants of the source sentence are created, each containing a "cue" that forces a specific interpretation (e.g., adding "female" or "male" markers).
  3. Scoring: A separate, high-quality evaluator model scores the original translation against these modified sources.
Examples of contrastive conditioning for lexical (left) and gender (right) disambiguation.

This approach allows for scalable, reference-free evaluation of how models handle ambiguous terms like "doctor" (gender) or "starter" (lexical sense). Validation of the method showed high agreement with human annotators (up to 97.6% for weighted English-German tasks), proving it is a robust surrogate for human evaluation.

Results: The Impact of Distillation

The study demonstrates a clear link between distillation and increased bias. Multiple regression analysis isolated SeqKD as a significant factor in reducing disambiguation accuracy. Notably, while student models often achieve competitive BLEU scores, they consistently perform worse on minority cases.

In gender-related tasks, the "Original Corpus" produced female forms for "doctor" in 2.0% of instances. This frequency dropped to 1.4% in the "Teacher" model and further plummeted to 1.1% in the "Student" model. This suggests that distillation compounds existing biases in the training data.

Percentage of male vs. female translations for 'doctor' across different model types.

Discussion and Conclusion

The findings highlight a critical trade-off in MT development: efficiency versus adequacy. Standard metrics like BLEU are "blind" to these specific failures, as they reward overall fluency while ignoring the systematic erasure of minority linguistic forms. The authors advocate for the use of "minimum accuracy" metrics—measuring performance on the worst-performing categories—to better capture these biases. Future work should explore diversity-preserving distillation techniques to ensure that the benefits of model compression do not come at the expense of linguistic fairness and accuracy.

Summary updated

BibTeX
@inproceedings{vamvas2021contrastive,
  author = {Jannis Vamvas and Rico Sennrich},
  title = {{Contrastive Conditioning for Assessing Disambiguation in MT: A Case Study of Distilled Bias}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.803},
}
Figure