Contextual Label Projection for Cross-Lingual Structured Prediction

Tanmay Parekh, I-Hung Hsu, Kuan-Hao Huang, Kai-Wei Chang, Nanyun Peng

TL;DR

The authors introduce CLaP (Contextual Label Projection), a novel method that uses instruction-tuned Large Language Models (LLMs) to accurately project labels across languages for structured prediction tasks. By treating the translated sentence as context for label translation, CLaP significantly outperforms traditional alignment and marker-based methods, yielding up to a 2.4 F1 improvement in zero-shot cross-lingual transfer across 39 languages.

What Problem Does This Paper Address?

Cross-lingual transfer is a vital technique for building Natural Language Processing (NLP) tools for the thousands of "low-resource" languages that lack massive labeled datasets. A popular approach is translate-train: you take your high-quality English training data, translate the sentences into a target language (like Swahili or Chinese), and train a model on this new "pseudo-data."

However, for structured prediction tasks—like Named Entity Recognition (NER) or Event Argument Extraction (EAE)—simply translating the text isn't enough. You also need to know exactly where the labels (e.g., "Person," "Defendant," "Location") ended up in the translated sentence. This process is called label projection.

Existing methods struggle with a trade-off:

  1. Marker-based methods insert tags (like [0]South Florida[/0]) into the source text before translation. This often confuses machine translation engines, leading to garbled output.
  2. Word-alignment methods translate the sentence normally and then try to map words back to the source using external tools. These often fail to capture the correct sense of a word (e.g., translating "suits" as "clothing" instead of "lawsuits") or fail to find the exact span in the translated text.
Figure 1: Illustration of the task of label projection from English to Chinese. Label projection converts sentences from a source to a target language while translating the associated labels jointly. Failures in this process occur when (a) labels are either inaccurately translated or (b) missing in the translated sentence in the target language.

Methods

The proposed method, CLaP, breaks the problem into two distinct, cleaner steps:

  1. Sentence Translation: The source sentence is translated into the target language using a standard, "plug-and-play" machine translation model (like Google Translate). This ensures the highest possible translation quality for the text itself.
  2. Contextual Label Translation: Instead of using heuristics to find the labels, CLaP uses an instruction-tuned LLM (like Llama-2-13B or GPT-3.5). The LLM is given the translated sentence as context and asked to provide the correct translation for specific source labels as they appear in that specific sentence.

By providing the translated sentence as context, the LLM can disambiguate words with multiple meanings and ensure the translated label is "faithful"—meaning it actually exists within the translated sentence.

Comparison of Marker-based, Word Alignment, and the proposed CLaP method.

Datasets

The authors benchmarked CLaP on two representative tasks across a wide array of languages:

  • Event Argument Extraction (EAE): Using the ACE dataset (multilingual).
  • Named Entity Recognition (NER): Using WikiANN (covering 39 languages) and MasakhaNER (focused on 10 extremely low-resource African and South American languages, such as Yoruba, Quechua, and Kinyarwanda).

Key Insights and Ideas

  • Context is King: The "aha moment" is that label translation shouldn't happen in a vacuum. The translated sentence provides the necessary semantic constraints to ensure the label's translation is both accurate in meaning and present in the text.
  • LLMs as Data Generators, Not Just Predictors: A major finding is that using an LLM to project labels to create training data for a smaller, specialized model (like XLM-R) works significantly better than asking the LLM to perform the task directly (zero-shot inference).
  • Accuracy vs. Faithfulness: The paper highlights two metrics for label projection. Accuracy (is the translation correct?) and Faithfulness (is the label in the sentence?). CLaP strikes the best balance, whereas older methods often sacrifice accuracy for faithfulness.
Intrinsic evaluation showing CLaP provides the best balance of accuracy and faithfulness.

Key Results

  • Performance Gains: CLaP achieved an average improvement of 2.4 F1 points for EAE and 1.4 F1 points for NER over the best existing baselines across 39 languages.
  • Low-Resource Success: On extremely low-resource languages, CLaP showed strong generalization. For instance, using a larger LLM (GPT-3.5) for the projection step led to improvements of 3 to 30 F1 points for languages like Urdu and Kazakh.
  • Qualitative Superiority: Human evaluations confirmed that CLaP produces much more natural and contextually appropriate label translations compared to word-alignment tools, which often produce "extra words" or "incomplete" translations.

Limitations and Future Work

  • Language Bias: The authors note that the underlying LLMs (like Llama-2) are English-centric. Their performance varies significantly across languages depending on how much data for that language was in the LLM's training set.
  • Computational Cost: Using LLMs for label projection is more computationally expensive than simple word alignment, especially when scaling to millions of sentences.
  • Future Directions: The authors suggest that as multilingual LLMs continue to grow in size and capability, the effectiveness of CLaP will likely improve further, particularly for the world's most under-represented languages.

Summary updated

Abstract

Label projection, which involves obtaining translated labels and texts jointly, is essential for leveraging machine translation to facilitate cross-lingual transfer in structured prediction tasks. Prior research exploring label projection often compromise translation accuracy by favoring simplified label translation or relying solely on word-level alignments. In this paper, we introduce a novel label projection approach, CLaP, which translates text to the target language and performs contextual translation on the labels using the translated text as the context, ensuring better accuracy for the translated labels. We leverage instruction-tuned language models with multilingual capabilities as our contextual translator, imposing the constraint of the presence of translated labels in the translated text via instructions. We benchmark CLaP with other label projection techniques on zero-shot cross-lingual transfer across 39 languages on two representative structured prediction tasks - event argument extraction (EAE) and named entity recognition (NER), showing over 2.4 F1 improvement for EAE and 1.4 F1 improvement for NER. We further explore the applicability of CLaP on ten extremely low-resource languages to showcase its potential for cross-lingual structured prediction.
BibTeX
@inproceedings{parekh2024contextual,
  author = {Tanmay Parekh and I-Hung Hsu and Kuan-Hao Huang and Kai-Wei Chang and Nanyun Peng},
  title = {{Contextual Label Projection for Cross-Lingual Structured Prediction}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.321},
}
Figure