Translate and Classify: Improving Sequence Level Classification for English-Hindi Code-Mixed Data

Devansh Gautam, Kshitij Gupta, Manish Shrivastava

Abstract

Code-mixing, the practice of alternating between languages in a single conversation, is a dominant mode of communication on social media, yet it remains a significant challenge for Natural Language Processing (NLP). This paper presents a "Translate and Classify" strategy for English-Hindi code-mixed (Hinglish) data. By leveraging a fine-tuned mBART model to translate code-mixed text into English and subsequently applying state-of-the-art English classifiers, the researchers achieved a 12.4% absolute improvement in Natural Language Inference (NLI) and a 5.3% boost in Sentiment Analysis over previous benchmarks.

Introduction

For millions of social media users in India, "Hinglish"—a blend of Hindi and English—is the primary language of digital interaction. However, developing robust NLP models for code-mixed data is hampered by three primary factors:

  1. Data Scarcity: Annotated code-mixed datasets are significantly smaller than monolingual English or Hindi corpora.
  2. Informal Structure: Hinglish typically uses Roman script with non-standard, phonetic spellings, complicating tokenization and parsing.
  3. Model Limitations: Multilingual models like mBERT often struggle with the "noise" of code-switching in complex reasoning tasks.

The authors investigate whether translating code-mixed text into a high-resource language like English can bypass these hurdles and unlock the power of advanced English-only classifiers.

Methodology

The proposed pipeline consists of three distinct stages designed to handle the noise of code-mixed input and transform it into a format suitable for high-performance classification.

The proposed Translate and Classify pipeline, illustrating the flow from raw Hinglish to sentiment classification.

1. Transliteration and Normalization

Since Hinglish is predominantly written in Roman script, the first step involves using the CSNLI library to transliterate Hindi words into Devanagari script. This normalization step reduces vocabulary noise and prepares the text for the translation model.

2. Translation

The core of the pipeline is a translation module based on mBART, a multilingual sequence-to-sequence model. The researchers employed a hybrid transfer learning approach (mBART-hien-cm), where the model was first trained on standard Hindi-English parallel sentences and then fine-tuned on a smaller set of code-mixed-to-English pairs. This method proved essential for capturing the unique syntax of Hinglish.

3. Classification

Once translated into English, the text is processed by specialized English-only models. For NLI, the authors evaluated RoBERTa, XLNet, ALBERT, and DeBERTa. For Sentiment Analysis, they utilized models pre-trained on social media data, such as BERTweet and RoB-RT.

Experimental Setup and Datasets

The approach was validated using several benchmark datasets:

  • Translation: Data from Dhar et al. (2018) and Srivastava and Singh (2020) for code-mixed text, alongside the IIT Bombay English-Hindi corpus for monolingual data.
  • Natural Language Inference (NLI): The English-Hindi NLI dataset from the GLUECoS benchmark, derived from Bollywood movie scripts.
  • Sentiment Analysis: The English-Hindi Sentiment Analysis dataset from GLUECoS, consisting of code-mixed tweets.

Key Results

The "Translate and Classify" strategy yielded substantial performance gains across all tasks:

  • Natural Language Inference: The RoBERTa-large model achieved 75.53% accuracy, representing a 12.4% absolute increase over the previous state-of-the-art (SOTA) of 63.1%.
  • Sentiment Analysis: The pipeline reached a 64.6 F1-weighted score, a 5.3% absolute increase over the previous best of 59.35%.
  • Translation Quality: The best mBART configuration achieved BLEU scores of 31.7 and 33.0, indicating successful mapping from noisy code-mixed input to coherent English.

Discussion and Limitations

A critical insight from this work is that the advantages of using superior English classifiers—trained on massive datasets—far outweigh the errors introduced during translation. However, the pipeline is not without its challenges.

Confusion matrix for Sentiment Analysis showing that the model frequently misclassifies negative sentiment as neutral.

As shown in the confusion matrix above, the sentiment analysis model struggled specifically with negative sentiment, misclassifying negative tweets as neutral in approximately 37% of cases. This suggests that certain cultural nuances or sarcasm may be lost in translation. Furthermore, the system's overall performance is heavily dependent on the quality of the initial translation; if the translator fails to capture the core meaning, the classifier cannot recover.

Conclusion

The "Translate and Classify" framework demonstrates that for low-resource code-mixed languages, translation into a high-resource language can be a more effective strategy than building specialized code-mixed models from scratch. Future work will focus on synthetic data generation to expand parallel corpora and extending this framework to other code-mixed language pairs.

Summary updated

BibTeX
@inproceedings{gautam2021translate,
  author = {Devansh Gautam and Kshitij Gupta and Manish Shrivastava},
  title = {{Translate and Classify: Improving Sequence Level Classification for English-Hindi Code-Mixed Data}},
  booktitle = {CALCS @ NAACL},
  year = {2021},
  doi = {10.18653/v1/2021.calcs-1.3},
}
Figure