SSP: Self-Supervised Prompting for Cross-Lingual Transfer to Low-Resource Languages using Large Language Models

Vipul Kumar Rathore, Aniruddha Deb, Ankish Kumar Chandresh, Parag Singla, MausamEMNLP 2024 · DOI

Abstract

Large Language Models (LLMs) like GPT-4 exhibit a significant performance gap when processing Low-Resource Languages (LRLs) compared to high-resource ones. We present Self-Supervised Prompting (SSP), a novel two-stage framework for Zero-labeled Cross-Lingual Transfer (0-CLT). SSP generates noisy "silver" labels in the target language and subsequently uses these samples as in-context exemplars. Our findings reveal that LLMs are remarkably robust to label noise if the language matches the target task, allowing SSP to consistently outperform state-of-the-art fine-tuned models and standard cross-lingual prompting methods.

Introduction

The challenge of Zero-labeled Cross-Lingual Transfer (0-CLT) involves performing tasks in a Low-Resource Language (LRL) using labeled data only from a different, often medium-resource language. Standard approaches, such as fine-tuning on a source language or prompting with English examples, frequently fail because the linguistic distance is too great for the model to bridge in a single step. SSP addresses this by turning noisy, automatically generated data into a powerful learning signal, prioritizing language structure over label perfection.

Methods

The SSP framework utilizes a two-stage "bootstrap and refine" pipeline to bridge the cross-lingual gap.

The SSP framework pipeline, illustrating the two-stage process from noisy labeling to refined in-context learning.

Stage I: Noisy Labeling

In the first stage, a source-language model (either a fine-tuned smaller model or the LLM itself prompted with source-language examples) generates initial "noisy" labels for the target LRL test set.

Stage II: In-Language In-Context Learning (ICL)

These noisy labeled examples are then used as In-Context Learning (ICL) exemplars for the LLM. By providing examples in the actual target language, the model can better grasp the specific syntax and morphology required for the task, even if some labels are incorrect.

Exemplar Selection via ILP

To optimize Stage II, the researchers developed a selection method based on Integer Linear Programming (ILP). The solver optimizes for three criteria:

  • Semantic Similarity: Selecting examples mathematically similar to the query.
  • Label Coverage: Ensuring the prompt contains at least one example of every possible label.
  • Prediction Confidence: Prioritizing examples where the Stage I model was highly confident.

Results and Key Insights

SSP was evaluated across eleven low-resource languages in tasks including POS tagging, Named Entity Recognition (NER), and Natural Language Inference (NLI). It achieved significant gains, such as a jump from 52.2 to 71.1 F1 in African NER tasks.

The "Noise Tolerance" Discovery

A pivotal insight from this work is the model's resilience to label noise when the language context is correct.

Performance (F1) vs. Label Noise (1-x), showing that prompting with noisy target-language data can outperform source-language baselines.

As illustrated in the analysis of label noise, there is a substantial "noise region" where prompting with imperfect target-language data outperforms prompting with 100% accurate source-language data. This suggests that seeing the target language's structure helps the LLM more than seeing perfect labels in a language it already knows well.

Conclusion

SSP demonstrates that for low-resource NLP, language-specific structure is often more valuable than label accuracy. While the two-stage process increases inference costs and faces challenges with high-fertility tokenization in some LRLs, it provides a robust and effective path for expanding LLM capabilities to underserved languages without requiring human-labeled data.

Summary updated

BibTeX
@inproceedings{rathore2024ssp,
  author = {Vipul Kumar Rathore and Aniruddha Deb and Ankish Kumar Chandresh and Parag Singla and Mausam},
  title = {{SSP: Self-Supervised Prompting for Cross-Lingual Transfer to Low-Resource Languages using Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.886},
}
Figure