PRO-CS : An Instance-Based Prompt Composition Technique for Code-Switched Tasks

Srijan Bansal, Suraj Tripathi, Sumit Agarwal, Teruko Mitamura, Eric NybergEMNLP 2022 · OpenReview

Abstract

Code-switching (CS)—the practice of alternating between two or more languages within a single conversation—is a global phenomenon that poses significant challenges for Natural Language Processing (NLP). Annotated datasets for CS are scarce, making standard fine-tuning (FT) prone to overfitting. We introduce PRO-CS (Instance-Based PROmpt Composition for Code-Switching), a parameter-efficient technique that dynamically composes specialized prompts for every input instance. By using an attention mechanism to combine knowledge from a library of monolingual task and language prompts, PRO-CS matches or exceeds full fine-tuning performance while training only 0.18% of the model's parameters.

Introduction

Building AI models for code-switched text is notoriously difficult. The standard approach of fine-tuning requires updating millions of parameters, which often leads to poor generalization when data is limited. While Prompt-Tuning (PT) offers a more efficient alternative by keeping the base model frozen and learning a small "soft prompt," it often struggles with the dual complexity of multi-lingual grammars and task-specific requirements inherent in code-switching.

Methodology

PRO-CS addresses these challenges by building a custom prompt for every sentence through three main components:

  1. Source Prompts: A "library" of prompts is pre-trained on large, monolingual datasets. This includes Language Prompts (capturing linguistic features) and Task Prompts (capturing task-specific logic like sentiment analysis or POS tagging).
  2. Target Prompt Initialization: To provide an inductive bias for the "switching" logic, a new prompt is initialized using a Language Identification (LID) task. This helps the model identify boundaries between languages.
  3. Instance-Based Composition: For every input sentence, an attention module evaluates the library of source prompts to determine their relevance. For instance, a sentence with a high proportion of Spanish will receive a higher weight for the Spanish language prompt. These weighted prompts are then combined into a single "composed" prompt.
Overview of PRO-CS composition scenarios: (a) standard prompt composition, (b) cross-lingual transfer, and (c) cross-task transfer.

The underlying multilingual model (e.g., mBERT) remains frozen throughout this process. Only the tiny prompts and the attention modules are trained, ensuring extreme parameter efficiency.

The architecture of the Task Attention module (G_t), which dynamically weights source prompts based on the input instance.

Experimental Setup and Datasets

The researchers evaluated PRO-CS on 10 datasets across 4 language pairs: English-Hindi (En-Hi), English-Spanish (En-Es), English-Tamil (En-Ta), and English-Bengali (En-Bn). The tasks included Sentiment Analysis (SA), Hate Speech detection (HS), Intent Classification (IN), Part-of-Speech tagging (POS), and Named Entity Recognition (NER). Monolingual sources like Wikipedia and standard benchmarks (e.g., IMDB, Penn Treebank) were used to build the initial prompt library.

Key Results

  • Massive Efficiency: PRO-CS utilizes only 314K trainable parameters (0.18% of the total model), a stark contrast to the 178M parameters updated during full fine-tuning.
  • Performance Gains: It outperformed standard Prompt-Tuning (PT) by an average of 3 to 3.5 points in Macro-F1 scores across all datasets.
  • Competitive with Fine-Tuning: Despite the massive reduction in trainable parameters, PRO-CS matched or beat full fine-tuning on several tasks, such as En-Es Sentiment Analysis and En-Hi POS tagging.
  • Low-Resource Robustness: In simulations with only 512 training examples, PRO-CS proved significantly more stable than fine-tuning, which frequently overfit or failed to converge.

Conclusion

PRO-CS demonstrates that modular knowledge—represented as task and language prompts—can be effectively "mixed and matched" to solve complex code-switching tasks. While the model showed some limitations in capturing rare labels in imbalanced tasks like NER, its success in classification and sequence tagging marks a significant step forward for low-resource NLP. Future work will explore extending this composition technique to generative tasks such as machine translation and summarization.

Summary updated

BibTeX
@inproceedings{bansal2022pro,
  author = {Srijan Bansal and Suraj Tripathi and Sumit Agarwal and Teruko Mitamura and Eric Nyberg},
  title = {{PRO-CS : An Instance-Based Prompt Composition Technique for Code-Switched Tasks}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=TxManQqCix},
}
Figure