Investigating Multi-Pivot Ensembling with Massively Multilingual Machine Translation Models

Alireza Mohammadshahi, Jannis Vamvas, Rico SennrichNAACL 2024

TL;DR

Massively multilingual translation models struggle with low-resource languages. This paper explores multi-pivot ensembling to improve translation quality and reduce hallucinations, proposing a novel "MaxEns" strategy that selects the most confident pivot prediction. MaxEns shows promise, outperforming simple averaging and reducing hallucinations, though it still lags behind using English as a single pivot.

What Problem Does This Paper Address?

Modern neural machine translation (NMT) has seen remarkable progress with massively multilingual models, capable of translating between many languages with a single model. However, these models often falter on low-resource language pairs, where training data is scarce. A common workaround is "pivoting," where translation occurs indirectly through a high-resource language (like English). This paper investigates extending this by using multiple pivot languages simultaneously, a technique called multi-pivot ensembling. While promising, previous work using simple averaging of pivot predictions can actually increase undesirable "hallucinations" (generating text that is not present in the source) and underperform compared to a single, high-resource pivot. The authors aim to understand why this happens and propose a better way to combine multiple pivot translations.

Methods

The core of the paper lies in exploring and proposing new ensembling strategies for multi-pivot translation.

  • MultiAvg (Multilingual Average): This is the baseline multi-pivot strategy, inspired by prior work. It involves averaging the probability distributions of the predicted tokens across all chosen pivot languages.
  • MaxEns (Maximum Ensemble): This is the novel contribution. Instead of averaging probabilities, MaxEns selects the token with the highest probability from the most confident pivot prediction for each step of the translation. The intuition is that more confident predictions are less likely to be hallucinations.

The authors evaluate these methods by comparing them against direct translation and single-pivot translation (specifically using English).

Datasets

  • FLORES-101 benchmark: This dataset was used for evaluation. It contains 3,001 sentences translated into 101 languages by humans. The authors specifically focused on a subset of 20 low-resource language directions from FLORES-101, choosing languages from different Indo-European branches (Germanic, Romance, Slavic, Indo-Aryan, and Iranian).

Key Insights and Ideas

  • Hallucinations are "sticky": The paper highlights a finding that hallucinations can be similar across different translation paths, especially when models share similar architectures and training data. This "stickiness" can be exacerbated by simple averaging of pivot predictions.
  • Confidence as a hallucination heuristic: The MaxEns strategy is motivated by the idea that model confidence can be a good indicator of translation quality and a predictor of hallucinations. Higher confidence predictions are hypothesized to be less prone to generating incorrect or fabricated content.
  • English remains a strong pivot: Despite exploring multi-pivot strategies, the results consistently show that using English as a single pivot often still yields better results, especially in terms of reducing hallucinations. This suggests that the sheer volume of English-centric data in massively multilingual models gives it a significant advantage.
  • Optimal strategy is direction-dependent: The paper concludes that the best pivoting strategy isn't universal. It varies significantly depending on the specific language pair, suggesting a need for future research into automatically identifying the most effective strategy for each direction.

Key Results

The authors evaluated their methods using two massively multilingual NMT models: M2M100 and SMaLL100.

  • MultiAvg vs. Direct Translation: The MultiAvg approach generally improved translation quality (measured by spBLEU) and reduced hallucinations compared to direct translation for the tested low-resource directions.
  • MultiAvg vs. English Pivot: However, MultiAvg consistently lagged behind using English as a single pivot, both in terms of translation quality and, more significantly, in hallucination reduction.
  • MaxEns vs. MultiAvg: MaxEns significantly outperformed the MultiAvg strategy. It achieved better translation performance and a notable reduction in hallucinations.
  • MaxEns vs. English Pivot: MaxEns achieved competitive translation quality with English pivoting on average. However, it still underperformed English pivoting in terms of hallucination reduction. For example, on average, MaxEns had a higher percentage of hallucinations (18.8%) compared to English pivoting (7.3%) for the M2M100 model.

Limitations and Future Work

  • Limited language pairs: The experiments were conducted on a subset of 20 low-resource language directions. Future work could expand this to more language directions, including medium-resource pairs.
  • Specific models: The study used two specific massively multilingual NMT models (M2M100 and SMaLL100). Extending the evaluation to newer, larger models like NLLB200 or large language models (LLMs) would be valuable.
  • Optimal strategy identification: The paper highlights the need for future research to develop methods for automatically determining the most effective translation strategy (direct, single-pivot, or multi-pivot) for any given language pair, without relying on development data.
  • Combining with other techniques: The authors suggest that future work could explore multi-pivot ensembling in combination with other techniques, such as source-contrastive decoding, which has shown promise in mitigating hallucinations.

Summary updated

BibTeX
@inproceedings{mohammadshahi2024investigating,
  author = {Alireza Mohammadshahi and Jannis Vamvas and Rico Sennrich},
  title = {{Investigating Multi-Pivot Ensembling with Massively Multilingual Machine Translation Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure