Few-Shot Learning Translation from New Languages

Carlos Mullov, Alexander WaibelEMNLP 2025

TL;DR

This paper investigates how much monolingual data is needed to train high-quality word representations for low-resource languages, enabling effective few-shot machine translation. They demonstrate that even with limited parallel data, good translation performance can be achieved if word embeddings are well-trained, but highlight the critical role of high-quality bilingual dictionaries for alignment.

What Problem Does This Paper Address?

The field of machine translation has seen significant progress with neural networks, particularly in leveraging transfer learning from high-resource languages to low-resource ones. However, a key assumption is the availability of high-quality word representations for the target language. This paper questions how much data is truly required to learn these representations and whether this is the primary bottleneck for low-resource language translation. They aim to understand the data requirements for effective word embedding training and its impact on few-shot translation, especially when parallel data is scarce.

Methods

The core of the methodology revolves around training word embeddings using the Continuous Bag-of-Words (CBOW) objective, incorporating subword information via fastText. These embeddings are then aligned to a common English embedding space using bilingual dictionaries. This aligned embedding space is then plugged into a pre-trained Transformer sequence-to-sequence machine translation model. The authors explore the impact of varying amounts of monolingual data used for training these word embeddings. For few-shot learning, they employ a fixed setup of fine-tuning the NMT model on a small amount of parallel data (500 sentences) using low-rank adapters.

Datasets

  • Parallel Data: Used for pre-training the machine translation model and for few-shot fine-tuning. This includes ParaCrawl v9, Tatoeba, OPUS-100, TED2020, News-Commentary, MIZAN, GOURMET, Bianet, ELRC-EMEA, ELRC-4248-NTEU_TierA, CCAligned, OpenSubtitles, IndicTrans2, and HPLT. For evaluation, the Flores evaluation dataset (devtest split) and the Tatoeba challenge v2023-09-26 test split were used.
  • Monolingual Data: Used for training word representations. The paper compares several massively multilingual language modeling corpora:
  • MADLAD-400 (clean and noisy splits)
    • Fineweb 2
    • HPLT
    • CulturaX
  • Bilingual Dictionaries: Used for aligning word embeddings. The MUSE project dictionaries were used where available, and the Panlex project dictionaries were used for remaining languages. A curated high-quality Waray-English dictionary was also used for a specific case study.

Key Insights and Ideas

  • Word Embeddings are Key: The paper strongly suggests that high-quality word representations are crucial for effective few-shot translation, even with very limited parallel data.
  • Data Efficiency of CBOW: The CBOW objective, when trained on abundant monolingual data, can yield embeddings that enable rapid adaptation to new languages.
  • The Dictionary Bottleneck: The quality of bilingual dictionaries for aligning embedding spaces is a significant bottleneck. Noisy or incomplete dictionaries lead to substantially lower translation performance.
  • Monolingual Data Threshold: For many languages, around 10 million sentences of monolingual data seem to be a sweet spot, after which gains in translation quality diminish. However, some languages require significantly more data to reach even a basic cross-lingual signal.
  • Dataset Quality Matters: The paper highlights issues with the quality of publicly available multilingual language modeling datasets, particularly for languages like Waray, where excessive noise and irrelevant content hinder effective representation learning.

Key Results

  • With 23 million sentences of monolingual data and only 500 parallel sentences for fine-tuning, the model achieved 19.7 BLEU on Flores for Tagalog.
  • In a simulated low-resource setting, using 31,250 sentences of monolingual data and 500 parallel sentences, the model exceeded 15 BLEU on Flores for several unseen languages.
  • For some languages, even with over 10 million sentences of monolingual data, translation scores remained below 5 BLEU, indicating issues beyond word representation quality.
  • The use of high-quality MUSE dictionaries for alignment consistently outperformed noisy Panlex dictionaries, leading to significant BLEU score improvements.

Limitations and Future Work

  • Spoken-only Languages: The current method relies on text data. Future work should explore adapting the method to acoustic word embeddings for spoken-only languages.
  • CBOW vs. Language Modeling Objectives: The paper hypothesizes CBOW is more data-efficient but doesn't empirically prove it against Transformer-based language modeling objectives. This comparison is left for future work.
  • English-centric Evaluation: The MT model is trained to translate into English. Translating into other languages would require additional fine-tuning.
  • Dictionary Quality: The reliance on high-quality bilingual dictionaries remains a significant challenge. Future work could explore better methods for inducing or improving these dictionaries.
  • Dataset Curation: The findings on Waray highlight the need for more sophisticated deduplication pipelines and diverse text sources for multilingual language modeling datasets.

Summary updated

BibTeX
@inproceedings{mullov2025few,
  author = {Carlos Mullov and Alexander Waibel},
  title = {{Few-Shot Learning Translation from New Languages}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure