Abstract
Code-switching, the practice of alternating between languages in a single conversation, is prevalent on social media but challenging for Natural Language Processing (NLP). This paper introduces a "late fusion" approach for sentiment analysis of code-switched data. By aggregating the output logits of multiple pre-trained transformer models through a secondary neural network, the proposed architecture achieves significant performance gains. On the Microsoft GLUECoS benchmark, the model reached an F1 score of 73.66% for English-Spanish and 61.24% for English-Hindi, outperforming standard multilingual baselines.
Introduction: The Challenge of Code-Switching
Social media users frequently mix languages, creating "noisy" linguistic environments that traditional monolingual NLP models cannot handle. Even multilingual models struggle with the informal syntax and lack of standardization inherent in code-switched text.
A primary hurdle is Multiple Romanization. In language pairs like English-Hindi, users often write Hindi words using the Roman alphabet. Because there is no standard spelling, a single word can have numerous variations, which confuses models relying on fixed vocabularies.
Methodology: Late Fusion Architecture
The core innovation of this work is the Late Fusion architecture. Rather than selecting a single best-performing model, the researchers combine the strengths of various pre-trained transformers, such as BERT, mBERT, XLM-R, and Hi-BERT.
The Fusion Process
- Base Models: Multiple state-of-the-art transformers are fine-tuned on the target code-switched datasets.
- Logit Extraction: For each input sentence, the raw, unnormalized prediction values (logits) for each sentiment class (positive, negative, neutral) are extracted from two different base models.
- Fusion Layer: These six logits are fed into a Multi-Layer Perceptron (MLP) with a single hidden layer of six neurons.
This "unison model" learns to weight the predictions of the base models, identifying which model is more reliable for specific linguistic patterns. By using logits instead of simple class labels, the fusion layer captures richer information about the models' confidence levels.
Experimental Setup and Datasets
The study utilizes the Microsoft GLUECoS (Generalized Language Understanding Evaluation for Code-Switched sequences) benchmark, focusing on two distinct language pairs:
- English-Spanish (En-Es): 1,523 training tweets representing closely related languages.
- English-Hindi (En-Hi): 10,080 training tweets representing largely different languages. For this pair, Devanagari script was transliterated into Roman script to maintain consistency.
Results and Discussion
The late fusion approach consistently outperformed individual models and previous benchmarks.
| Language Pair | Best Individual Model (F1) | Late Fusion Model (F1) | Improvement |
|---|---|---|---|
| English-Spanish | 69.42% (XLM-R) | 73.66% (XLM-R + mBERT) | +4.24% |
| English-Hindi | 59.05% (BERT) | 61.24% (mBERT + BERT) | +2.19% |
The English-Spanish result is particularly noteworthy, showing a significant improvement over the original mBERT baseline (63.02%) reported for GLUECoS. The fusion technique effectively reduces "prediction dispersion," making the system more robust to the spelling variations and slang common in social media data.
Conclusion and Future Work
Late fusion provides a powerful mechanism for enhancing sentiment analysis in multilingual contexts. By aggregating complementary views from different transformer architectures, the model achieves superior performance on noisy, code-switched data.
Future research will focus on extending this method to low-resource language pairs, such as English mixed with Dravidian languages, and optimizing the architecture to reduce the computational overhead associated with running multiple large-scale transformers.
Summary updated