Enhancing Small Language Models for Cross-Lingual Generalized Zero-Shot Classification with Soft Prompt Tuning

Fred Philippy, Siwen Guo, Cedric Lothritz, Jacques Klein, Tegawendé BissyandéNAACL 2025

TL;DR

This paper introduces RoSPrompt, a novel soft prompt tuning method that significantly improves cross-lingual zero-shot classification (ZSC) for small language models. By leveraging multilingual class descriptions and contrastive label smoothing, RoSPrompt achieves robust generalization across data distributions and languages, outperforming existing prompt-based methods.

What Problem Does This Paper Address?

Zero-shot classification (ZSC) is crucial for classifying text into categories unseen during training, especially in low-resource languages where labeled data is scarce. While pre-trained language models (PLMs) show promise, they often rely on extensive training data or external knowledge. Existing prompt-based ZSC methods, while reducing data dependence, struggle with cross-lingual transfer due to language-specific prompts and often fail to effectively utilize available labeled data from related tasks. This paper addresses the challenge of creating efficient, robust, and cross-lingually adaptable ZSC methods, particularly for smaller multilingual PLMs.

Methods

The core of this work is RoSPrompt, a hybrid approach combining Soft Prompt Tuning (SPT) and Nonparametric Prompting (NPPrompt).

  • Soft Prompt Tuning (SPT): Instead of manually crafting "hard" natural language prompts, RoSPrompt transforms them into trainable "soft" prompts. These are learned vectors appended to the input, allowing for data-driven adaptation.
  • Multilingual Class Description Tokens: To overcome language bias in prompts, RoSPrompt extends the traditional single-token verbalizer with a set of multilingual tokens that better capture class essence across languages.
  • Contrastive Label Smoothing: To prevent overfitting and improve generalization, a modified label smoothing technique is employed. This method distributes probabilities not only to the true class tokens but also to other tokens, while penalizing the model for consistently favoring class tokens over non-class tokens.
  • Penalty Term: An additional penalty is introduced to further discourage overfitting on seen classes during training. This penalty is based on the ratio of predicted probabilities for true class tokens versus other class tokens.
  • Inference: For inference, the paper adopts the aggregation technique proposed by Zhao et al. (2023), which efficiently combines model outputs for classification.

Datasets

  • Training Data: The English DBPedia14 dataset, an ontology classification dataset with 14 classes, was used to train the soft prompts.
  • Evaluation Datasets:
    • MLSUM: A multilingual news summarization dataset used to test robustness against data distribution shifts.
  • MTOP: A multilingual utterance classification dataset with 11 domains, used for testing robustness against data distribution shifts.
  • SIB-200: A multilingual topic classification dataset covering over 200 languages, used to assess cross-lingual transfer to low-resource languages.

Key Insights and Ideas

  • Soft prompts are superior to hard prompts for cross-lingual ZSC: Fine-tuning soft prompts, even with a small amount of data, leads to significantly better performance than using translated hard prompts.
  • Multilingual class descriptions are crucial: Moving beyond single-language verbalizers is essential for effective cross-lingual ZSC, as it allows models to better understand class semantics across different languages.
  • Contrastive label smoothing enhances generalization: This technique helps mitigate overfitting and improves the model's ability to generalize to unseen classes, especially in challenging cross-lingual scenarios.
  • Small models can achieve strong cross-lingual ZSC: RoSPrompt demonstrates that even smaller multilingual PLMs can achieve competitive or superior performance compared to larger models, particularly in low-resource settings.
  • Data efficiency is a major advantage: The method requires only a small number of labeled samples from a comparable classification task for training, making it highly practical.

Key Results

  • RoSPrompt consistently outperforms baselines like NPPrompt and SPT across various multilingual models (XGLM, mT0, XLM-R) and datasets (MTOP, MLSUM, SIB-200).
  • For instance, on the MTOP dataset, RoSPrompt achieved accuracy scores significantly higher than NPPrompt and SPT across different models.
  • On the SIB-200 dataset, RoSPrompt showed strong cross-lingual transfer, outperforming baselines, especially for language families where data is scarce.
  • The ablation study revealed that all components of RoSPrompt (multilingual labels, contrastive label smoothing, and penalty) are integral to its success, with the removal of multilingual labels having the greatest negative impact on XLM-R.
  • The method is computationally efficient, fine-tuning fewer than 1% of parameters.

Limitations and Future Work

  • Domain Limitations: The study was conducted on specific domains, and effectiveness across entirely distinct or niche domains remains to be explored.
  • Few-Shot Sample Selection Bias: While efforts were made to reduce bias, the specific selection of few-shot samples can still influence outcomes.
  • Generalization to other tasks: While the paper focuses on topic classification, the authors suggest the approach could be effective for other classification tasks.
  • Larger Models: While RoSPrompt targets smaller LLMs, its applicability to larger models with billions of parameters is noted as a potential future direction.

Summary updated

BibTeX
@inproceedings{philippy2025enhancing,
  author = {Fred Philippy and Siwen Guo and Cedric Lothritz and Jacques Klein and Tegawendé Bissyandé},
  title = {{Enhancing Small Language Models for Cross-Lingual Generalized Zero-Shot Classification with Soft Prompt Tuning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure