TL;DR
The authors introduce CLaP (Contextual Label Projection), a novel method that uses instruction-tuned Large Language Models (LLMs) to accurately project labels across languages for structured prediction tasks. By treating the translated sentence as context for label translation, CLaP significantly outperforms traditional alignment and marker-based methods, yielding up to a 2.4 F1 improvement in zero-shot cross-lingual transfer across 39 languages.
What Problem Does This Paper Address?
Cross-lingual transfer is a vital technique for building Natural Language Processing (NLP) tools for the thousands of "low-resource" languages that lack massive labeled datasets. A popular approach is translate-train: you take your high-quality English training data, translate the sentences into a target language (like Swahili or Chinese), and train a model on this new "pseudo-data."
However, for structured prediction tasks—like Named Entity Recognition (NER) or Event Argument Extraction (EAE)—simply translating the text isn't enough. You also need to know exactly where the labels (e.g., "Person," "Defendant," "Location") ended up in the translated sentence. This process is called label projection.
Existing methods struggle with a trade-off:
- Marker-based methods insert tags (like
[0]South Florida[/0]) into the source text before translation. This often confuses machine translation engines, leading to garbled output. - Word-alignment methods translate the sentence normally and then try to map words back to the source using external tools. These often fail to capture the correct sense of a word (e.g., translating "suits" as "clothing" instead of "lawsuits") or fail to find the exact span in the translated text.
Methods
The proposed method, CLaP, breaks the problem into two distinct, cleaner steps:
- Sentence Translation: The source sentence is translated into the target language using a standard, "plug-and-play" machine translation model (like Google Translate). This ensures the highest possible translation quality for the text itself.
- Contextual Label Translation: Instead of using heuristics to find the labels, CLaP uses an instruction-tuned LLM (like Llama-2-13B or GPT-3.5). The LLM is given the translated sentence as context and asked to provide the correct translation for specific source labels as they appear in that specific sentence.
By providing the translated sentence as context, the LLM can disambiguate words with multiple meanings and ensure the translated label is "faithful"—meaning it actually exists within the translated sentence.
Datasets
The authors benchmarked CLaP on two representative tasks across a wide array of languages:
- Event Argument Extraction (EAE): Using the ACE dataset (multilingual).
- Named Entity Recognition (NER): Using WikiANN (covering 39 languages) and MasakhaNER (focused on 10 extremely low-resource African and South American languages, such as Yoruba, Quechua, and Kinyarwanda).
Key Insights and Ideas
- Context is King: The "aha moment" is that label translation shouldn't happen in a vacuum. The translated sentence provides the necessary semantic constraints to ensure the label's translation is both accurate in meaning and present in the text.
- LLMs as Data Generators, Not Just Predictors: A major finding is that using an LLM to project labels to create training data for a smaller, specialized model (like XLM-R) works significantly better than asking the LLM to perform the task directly (zero-shot inference).
- Accuracy vs. Faithfulness: The paper highlights two metrics for label projection. Accuracy (is the translation correct?) and Faithfulness (is the label in the sentence?). CLaP strikes the best balance, whereas older methods often sacrifice accuracy for faithfulness.
Key Results
- Performance Gains: CLaP achieved an average improvement of 2.4 F1 points for EAE and 1.4 F1 points for NER over the best existing baselines across 39 languages.
- Low-Resource Success: On extremely low-resource languages, CLaP showed strong generalization. For instance, using a larger LLM (GPT-3.5) for the projection step led to improvements of 3 to 30 F1 points for languages like Urdu and Kazakh.
- Qualitative Superiority: Human evaluations confirmed that CLaP produces much more natural and contextually appropriate label translations compared to word-alignment tools, which often produce "extra words" or "incomplete" translations.
Limitations and Future Work
- Language Bias: The authors note that the underlying LLMs (like Llama-2) are English-centric. Their performance varies significantly across languages depending on how much data for that language was in the LLM's training set.
- Computational Cost: Using LLMs for label projection is more computationally expensive than simple word alignment, especially when scaling to millions of sentences.
- Future Directions: The authors suggest that as multilingual LLMs continue to grow in size and capability, the effectiveness of CLaP will likely improve further, particularly for the world's most under-represented languages.
Summary updated