Abstract
Multilingual speakers frequently employ code-switching, yet large-scale multilingual models often struggle with such data due to their primarily monolingual training. This paper proposes novel pretraining strategies to enhance performance on code-switched text in data-constrained scenarios. We introduce SWITCHMLM, an objective targeting language switch-points, and RESBERT, an architecture utilizing residual connections from intermediate layers to the Masked Language Modeling (MLM) head. Evaluation across four language pairs demonstrates significant improvements in Sentiment Analysis and Question Answering, highlighting the importance of structural awareness and intermediate layer representations in code-switched NLP.
Introduction
Multilingual speakers often engage in "code-switching," blending languages within a single conversation or sentence (e.g., "Laptop mere bag me rakha hai"). While massive multilingual models like mBERT and XLM-R exist, they struggle with code-switched tasks because they are primarily trained on monolingual data. The standard solution is "intermediate pretraining" on code-switched text before fine-tuning. However, high-quality, unlabeled code-switched text is scarce. This research tackles the challenge of making pretraining more "sample-efficient"—extracting the maximum possible linguistic signal from limited amounts of code-switched data.
Methodology
Modified Pretraining Objectives
Instead of standard Masked Language Modeling (MLM), which masks tokens at random, we propose:
- SWITCHMLM: This objective specifically targets "switch-points"—the tokens that straddle a language boundary. By forcing the model to predict words at these transitions, it learns the underlying structural patterns of how languages are mixed.
- FREQMLM: A surrogate method that uses relative word frequencies from large monolingual corpora to estimate language boundaries, making the technique applicable even when human-annotated Language Identification (LID) is missing.
Architectural Modifications (RESBERT)
We hypothesize that lower layers of models like mBERT capture more general linguistic and syntactic information, while higher layers become too task-specific. To leverage this, we introduce RESBERT.
As shown in the architecture diagram, RESBERT incorporates:
- Residual Connections: A regularized residual connection from an intermediate layer (Transformer block $x$) directly to the MLM head.
- Auxiliary LID Loss: An auxiliary loss function ($\mathcal{L}_{\text{aux}}$) that explicitly trains the intermediate layer to identify whether a token is a switch-point.
Experimental Setup
The study evaluates performance across four code-switched language pairs: Hindi-English, Spanish-English, Tamil-English, and Malayalam-English. Pretraining data was aggregated from various sources, totaling between 34K and 185K sentences per pair. Evaluation benchmarks included Sentiment Analysis (SA) and Question Answering (QA).
Results and Probing Analysis
The proposed techniques consistently outperformed standard pretraining (STDMLM). On Hindi-English QA, the best variant achieved a relative improvement of 5.8 F1 points over the baseline. Sentiment Analysis gains were also notable, such as a 2.7 F1 point increase for Tamil-English.
Probing experiments revealed several key insights:
- The Power of the Boundary: Masking at switch-points is more effective than random masking, forcing the model to learn the "grammar of switching."
- Lower Layers Matter: Probing revealed that intermediate layers (e.g., Layer 2 or 9) often encode more switch-point information than the final layer (Layer 12), as seen in the linear probing plots.
- Validation via Conditional Probing: Using "Conditional Probing," the authors demonstrated that their models carry more switch-point information in their neural representations compared to standard mBERT.
Conclusion
This work demonstrates that focusing on language boundaries and utilizing intermediate layer representations can significantly improve code-switched NLP performance in low-resource settings. Future work may explore automated layer selection via gating functions and scaling these techniques to larger datasets.
Summary updated