Zero-Shot vs. Translation-Based Cross-Lingual Transfer: The Case of Lexical Gaps

Abteen Ebrahimi, Katharina von der WenseNAACL 2024 · DOI

TL;DR

This paper investigates two primary methods for cross-lingual transfer in NLP: zero-shot learning and translation-based transfer. By introducing a novel dataset focused on "lexical gaps" (concepts with unique words in one language but not another), the authors find that while zero-shot methods currently outperform translation-based approaches due to poor machine translation quality, the latter holds greater long-term potential if translation accuracy can be improved.

What Problem Does This Paper Address?

Natural Language Processing (NLP) models often struggle to perform well across different languages. Cross-lingual transfer aims to bridge this gap, allowing models trained on one language to work on others. Two main strategies exist: zero-shot transfer, where a single multilingual model is fine-tuned on a source language and then applied directly to a target language, and translation-based transfer, where data in the target language is first translated into the source language (usually English) before being processed by a downstream model. While zero-shot has been dominant, its effectiveness, especially for low-resource languages, is being questioned. This paper explores which approach has more potential for high-level performance across all languages, focusing on a specific linguistic challenge: lexical gaps. These are concepts that have a single, dedicated word in one language but require multiple words or phrases to express in another. For example, Farsi distinguishes between paternal and maternal uncles ("amoo" vs. "daei"), while English uses the single term "uncle." This linguistic nuance poses a significant challenge for both zero-shot models and machine translation systems.

Methods

The authors introduce a new question-answering (QA) dataset specifically designed to evaluate performance on lexical gaps. The dataset is constructed using templates that create simple, factual sentences involving kinship terms. They then compare two main approaches:

  • Zero-Shot Transfer: They evaluate large language models (LLMs) like GPT-3.5, Llama 2, and BLOOM-Z in a zero-shot setting. These models are expected to understand and answer questions in a target language without explicit training on that language's specific data, relying on their multilingual pre-training.
  • Translation-Based Transfer: This involves translating the target language data into English. The paper compares two types of translation:
  • Machine Translation (MT): Using off-the-shelf MT systems like Google Translate and NLLB.
  • Oracle Translation: This represents a perfect, human-level translation, serving as an upper bound for translation-based methods.

They also conduct a human study to assess if human translators, without task-specific instructions, can produce translations that preserve the necessary nuance of lexical gaps.

Datasets

The primary dataset introduced in this paper is a novel QA dataset focused on lexical gaps. This dataset is created using templates and focuses on kinship terminology across five languages: Catalan, German, Farsi, Hindi, and Vietnamese. The dataset is designed to be simple for humans but challenging for NLP models, particularly when lexical gaps are involved.

Key Insights and Ideas

  • Lexical Gaps are a Tough Nut to Crack: Both zero-shot and translation-based methods struggle with lexical gaps, highlighting a significant linguistic challenge for NLP.
  • Zero-Shot is Currently Stronger, but Translation Has Potential: While current zero-shot models perform better on this specific task, the authors argue that translation-based methods, if paired with sufficiently accurate translations, could achieve superior long-term performance.
  • MT Quality is the Bottleneck: The primary reason translation-based methods lag behind is the inadequacy of current MT systems to capture the specific nuances of lexical gaps, often defaulting to general translations (e.g., translating both "amoo" and "daei" to "uncle").
  • Oracle Translation is the Gold Standard: Perfect translations (oracle translations) demonstrate the high potential of the translation-based approach, significantly outperforming zero-shot methods.
  • Human Translation is Mixed: While many human translators can preserve lexical gap information, not all do, especially without explicit task instructions. This suggests that task-specific translation data is crucial for effective MT.

Key Results

  • Zero-Shot Performance: For the tested LLMs (GPT-3.5, Llama 2, BLOOM-Z), zero-shot performance decreases when encountering lexical gap entities. BLOOM showed a 6.55% drop on average, while GPT-3.5 had a smaller drop of 3.12%.
  • Oracle Translation Advantage: Oracle translations consistently outperformed zero-shot approaches. The improvement was more pronounced for languages with greater divergence from English in their kinship terminology.
  • MT vs. Oracle: Machine translation (using NLLB and Google Translate) significantly underperformed oracle translations. For instance, GPT-3.5 on oracle Farsi data achieved 98.70% accuracy, while NLLB translation yielded only 66.60%. Google Translate performance dropped to random levels.
  • Human Study Findings: A majority of human translators (4 out of 5 in Group 1, 5 out of 6 in Group 2) naturally leaned towards specific translations for conflicting lexical gaps, supporting the "One Sense Per Discourse" hypothesis. However, this was more consistent when participants weren't shown non-conflicting examples first.

Limitations and Future Work

  • Dataset Simplicity: The dataset uses simple, template-based examples, which might not fully represent the complexity of real-world language. While this allows for fine-grained analysis, it could lead to models achieving near-perfect accuracy with finetuning, making zero-shot evaluation the focus.
  • Limited Languages: The study focused on only five languages, with only three exhibiting significant lexical gaps with English. Future work could expand to a more diverse set of languages.
  • MT Improvement is Key: The paper strongly suggests that improving MT quality is crucial for the success of translation-based cross-lingual transfer. This could involve collecting task-specific parallel data or exploring neuro-symbolic models.
  • Zero-Shot Potential: While current zero-shot models struggle, the authors acknowledge that future advancements in LLMs might enable them to better handle lexical gaps without explicit translation.
  • Human Translation Variability: The variability in human translation highlights the need for careful data collection for training MT models, ensuring translations are task-specific.

Summary updated

BibTeX
@inproceedings{ebrahimi2024zero,
  author = {Abteen Ebrahimi and Katharina von der Wense},
  title = {{Zero-Shot vs. Translation-Based Cross-Lingual Transfer: The Case of Lexical Gaps}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-short.37},
}
Figure