Abstract
Code-switching (CS)—the practice of alternating between two or more languages within a single conversation—is a global phenomenon that poses significant challenges for Natural Language Processing (NLP). Annotated datasets for CS are scarce, making standard fine-tuning (FT) prone to overfitting. We introduce PRO-CS (Instance-Based PROmpt Composition for Code-Switching), a parameter-efficient technique that dynamically composes specialized prompts for every input instance. By using an attention mechanism to combine knowledge from a library of monolingual task and language prompts, PRO-CS matches or exceeds full fine-tuning performance while training only 0.18% of the model's parameters.
Introduction
Building AI models for code-switched text is notoriously difficult. The standard approach of fine-tuning requires updating millions of parameters, which often leads to poor generalization when data is limited. While Prompt-Tuning (PT) offers a more efficient alternative by keeping the base model frozen and learning a small "soft prompt," it often struggles with the dual complexity of multi-lingual grammars and task-specific requirements inherent in code-switching.
Methodology
PRO-CS addresses these challenges by building a custom prompt for every sentence through three main components:
- Source Prompts: A "library" of prompts is pre-trained on large, monolingual datasets. This includes Language Prompts (capturing linguistic features) and Task Prompts (capturing task-specific logic like sentiment analysis or POS tagging).
- Target Prompt Initialization: To provide an inductive bias for the "switching" logic, a new prompt is initialized using a Language Identification (LID) task. This helps the model identify boundaries between languages.
- Instance-Based Composition: For every input sentence, an attention module evaluates the library of source prompts to determine their relevance. For instance, a sentence with a high proportion of Spanish will receive a higher weight for the Spanish language prompt. These weighted prompts are then combined into a single "composed" prompt.
The underlying multilingual model (e.g., mBERT) remains frozen throughout this process. Only the tiny prompts and the attention modules are trained, ensuring extreme parameter efficiency.
Experimental Setup and Datasets
The researchers evaluated PRO-CS on 10 datasets across 4 language pairs: English-Hindi (En-Hi), English-Spanish (En-Es), English-Tamil (En-Ta), and English-Bengali (En-Bn). The tasks included Sentiment Analysis (SA), Hate Speech detection (HS), Intent Classification (IN), Part-of-Speech tagging (POS), and Named Entity Recognition (NER). Monolingual sources like Wikipedia and standard benchmarks (e.g., IMDB, Penn Treebank) were used to build the initial prompt library.
Key Results
- Massive Efficiency: PRO-CS utilizes only 314K trainable parameters (0.18% of the total model), a stark contrast to the 178M parameters updated during full fine-tuning.
- Performance Gains: It outperformed standard Prompt-Tuning (PT) by an average of 3 to 3.5 points in Macro-F1 scores across all datasets.
- Competitive with Fine-Tuning: Despite the massive reduction in trainable parameters, PRO-CS matched or beat full fine-tuning on several tasks, such as En-Es Sentiment Analysis and En-Hi POS tagging.
- Low-Resource Robustness: In simulations with only 512 training examples, PRO-CS proved significantly more stable than fine-tuning, which frequently overfit or failed to converge.
Conclusion
PRO-CS demonstrates that modular knowledge—represented as task and language prompts—can be effectively "mixed and matched" to solve complex code-switching tasks. While the model showed some limitations in capturing rare labels in imbalanced tasks like NER, its success in classification and sequence tagging marks a significant step forward for low-resource NLP. Future work will explore extending this composition technique to generative tasks such as machine translation and summarization.
Summary updated