Power doesn’t reside in size: A Low Parameter Hybrid Language Model (HLM) for Sentiment Analysis in Code-mixed data

Pavan Sai Balaga, Nagasamudram Karthik, Challa Vishwanath, Raksha Sharma, Rudra Murthy, Ashish MittalEMNLP 2025

TL;DR

This paper introduces a Hybrid Language Model (HLM) that combines a powerful multilingual encoder with a lightweight decoder for sentiment analysis on code-mixed text. HLM achieves performance comparable to much larger Large Language Models (LLMs) while using significantly fewer parameters, making it a more efficient and accessible solution for real-world applications.

What Problem Does This Paper Address?

Code-mixed text, where multiple languages are used within a single utterance, is increasingly common in multilingual communities. Natural Language Processing (NLP) systems often struggle with this phenomenon due to the complex interplay of different grammatical structures. While fine-tuning large language models (LLMs) like GPT-3 or Llama on code-mixed data has shown promise, these models are computationally expensive and still lag behind their monolingual counterparts. This paper tackles the challenge of achieving high-performance sentiment analysis on code-mixed data with models that are both effective and resource-efficient, addressing the limitations of current LLMs in terms of computational cost and accessibility.

Methods

The core innovation is the Hybrid Language Model (HLM). This architecture fuses a multilingual encoder (e.g., mBERT, XLM-RoBERTa) with a lightweight decoder (e.g., Sarvam-1, Llama-3.2-1B). The encoder provides deep contextual understanding, while the decoder contributes generative priors. These two components are combined through a simple neural layer, and their pooled representations are fed into a feedforward classifier for sentiment prediction. This modular design allows for greater flexibility in combining heterogeneous components, unlike traditional fusion methods that typically combine similar model types. The paper also explores fine-tuning LLMs using the Q-LoRA framework for comparison.

Datasets

The paper evaluates HLM on four code-mixed sentiment analysis datasets:

  • ENG-HIN (English-Hindi): Romanized English-Hindi tweets.
  • ENG-SPA (English-Spanish): Tweets and CALCS workshop corpora.
  • ENG-TEL (English-Telugu): Tweets and YouTube comments.
  • ENG-MAL (English-Malayalam): YouTube comments.

These datasets are standard benchmarks for code-mixed NLP tasks.

Key Insights and Ideas

  • Size isn't everything: The research demonstrates that a well-designed, smaller model (HLM) can rival the performance of much larger LLMs on code-mixed sentiment analysis.
  • Hybridization is key: Combining the strengths of encoder and decoder architectures in a novel way leads to significant improvements.
  • Flexibility in fusion: HLM's ability to integrate diverse encoder and decoder components offers greater architectural freedom than traditional fusion methods.
  • Efficiency matters: The reduced parameter count of HLM translates to lower computational costs and better suitability for resource-constrained environments.

Key Results

  • HLM models, with as few as 1.5 to 3 billion parameters, achieve sentiment classification performance comparable to or better than LLMs with 7 to 9 billion parameters.
  • On the ENG-HIN dataset, HLM achieved an F1 score of 74.54, competitive with Gemma-2 9B (75.49) but using significantly fewer parameters.
  • For ENG-SPA, HLM outperformed Late Fusion across all configurations, with one setup reaching 58.69 F1.
  • On ENG-MAL, HLM surpassed individual models and even Gemma-2 9B in some configurations.
  • HLM consistently outperformed individual encoder and decoder models, as well as the Late Fusion baseline, across various configurations.
  • HLM models required significantly less GPU memory (24 GB) compared to large LLMs (40 GB) and trained faster per epoch.

Limitations and Future Work

The authors acknowledge that their work is limited to sentiment analysis due to computational resource constraints. Future work could explore the effectiveness of HLMs on other related tasks like emotion or sarcasm detection. Additionally, extending HLMs to handle sequence generation tasks is identified as an interesting and non-trivial challenge. The paper also touches upon the ethical considerations of deploying sentiment analysis systems on code-mixed data, highlighting potential biases and the need for community-based development practices.

Summary updated

BibTeX
@inproceedings{balaga2025power,
  author = {Pavan Sai Balaga and Nagasamudram Karthik and Challa Vishwanath and Raksha Sharma and Rudra Murthy and Ashish Mittal},
  title = {{Power doesn’t reside in size: A Low Parameter Hybrid Language Model (HLM) for Sentiment Analysis in Code-mixed data}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure