The Effects of Language Token Prefixing for Multilingual Machine Translation

Kevin Duh, Rachel Wicks, Kevin D. ShockleyAacl/ijcnlp 2022 · DOI

TL;DR

This paper investigates how different ways of "cueing" multilingual machine translation models with language information affect performance. They found that prepending language identifiers to the source sentence consistently improves translation quality, and the best token strategy can depend on the specific language set.

What Problem Does This Paper Address?

Modern machine translation systems are increasingly multilingual, handling many languages within a single model. To ensure the model translates to the correct target language, special "language tokens" are often added to the input or output sequences. However, there's a lot of variation in how these tokens are used: where they're placed (source vs. target side), whether they specify the source, target, or both, and if they're single tokens or pairs. This paper aims to systematically evaluate these choices in supervised translation settings, providing guidance for researchers and practitioners.

Methods

The authors trained bilingual Transformer models using various prefixing strategies outlined in Table 1 of the paper. They experimented with seven different combinations of source-side, target-side, and combined language tokens, and also explored the impact of encoder-decoder depth. Crucially, they focused on supervised translation (language pairs seen during training), differentiating their work from much prior research that focused on zero-shot scenarios. They used standard BLEU scores for evaluation and employed statistical significance testing to validate their findings.

Datasets

The experiments were conducted on two "small tracks" from the WMT21 Large-Scale Multilingual Shared Task.

  • TASK 1: Included Croatian (hr), Hungarian (hu), Estonian (et), Serbian (sr), Macedonian (mk), and English (en). This set features languages from both Indo-European and Uralic families, with a mix of Latin and Cyrillic scripts.
  • TASK 2: Included Javanese (jv), Indonesian (id), Malay (ms), Tagalog (tl), Tamil (ta), and English. This set is primarily composed of Malayo-Polynesian languages (written in Latin script), with Tamil being a Dravidian language in its own script. A combined dataset (TASK1,2) was also used. The paper notes that they did not upsample data for low-resource pairs to focus on the effect of prefixing strategies themselves.

Key Insights and Ideas

  • Source-side is king (for supervised translation): Prepending language tokens to the source sequence consistently yielded better or equal performance compared to target-side prefixing across various language sets.
  • Target token alone can be sufficient: While source-side prefixes generally help, the paper notes that in some cases, a target-only token (T|Ø) can perform competitively.
  • Language set matters: The optimal prefixing strategy isn't universal. For instance, a S2T|Ø (source-to-target pair) token performed exceptionally well on the combined TASK1,2 dataset but was outperformed by S|T|Ø and T|Ø on the individual tasks. This suggests that the diversity and balance of language pairs within a set influence which prefixing method is most effective.
  • Encoder depth helps, decoder depth less so: Increasing encoder depth improved performance for both source- and target-side prefixing strategies, while deeper decoders showed less benefit. This implies that the way the source information is processed is more critical for prefixing effectiveness.

Key Results

The paper presents detailed BLEU scores in Table 3 and Table 4. While specific numbers vary by task and prefixing strategy, the general trend is clear:

  • Source-side prefixes consistently outperformed target-side prefixes. For example, in TASK1, S|T|Ø and T|Ø were among the top performers, while target-side prefixes like Ø|S2T were generally lower.
  • The best-performing strategy in TASK1,2 was S2T|Ø, achieving scores around 25.2, significantly outperforming the best target-side prefix (Ø|S2T at 24.7).
  • In TASK2, source-side prefixes like S|T|Ø and T|Ø were significantly better than target-side options.
  • Table 4 shows that models with deeper encoders (e.g., 8 or 10 layers) generally performed better than shallower ones, regardless of the prefixing strategy.

Limitations and Future Work

The authors acknowledge that their work focuses on supervised settings and that upsampling was not performed, meaning the models might underperform on low-resource pairs. They suggest that future work should explore:

  • How prefixing strategies scale with larger and more diverse language sets.
  • More forceful prompting methodologies.
  • Further experimentation on how prefixes interact with language set scaling and diversity.
  • Investigating the trade-offs between performance gains and increased sequence length due to prefixes, especially concerning translation speed.

Summary updated

BibTeX
@inproceedings{duh2022effects,
  author = {Kevin Duh and Rachel Wicks and Kevin D. Shockley},
  title = {{The Effects of Language Token Prefixing for Multilingual Machine Translation}},
  booktitle = {Aacl/ijcnlp},
  year = {2022},
  doi = {10.18653/v1/2022.aacl-short.19},
}
Figure