Transfer Learning for Code-Mixed Data: Do Pretraining Languages Matter?

Kushal Tatariya, Heather Lent, Miryam de LhoneuxACL 2023

TL;DR

This paper investigates whether the languages used to pre-train large language models (LLMs) impact their performance on code-mixed data (e.g., mixing English and Hindi). They found that for standard fine-tuning, pre-training languages matter surprisingly little, but they significantly influence performance in zero-shot settings.

What Problem Does This Paper Address?

While LLMs have revolutionized NLP, they often struggle with code-mixed language, a common phenomenon in multilingual communities, especially on social media. Existing LLMs are typically trained on monolingual data, potentially leading to biases and suboptimal performance on mixed-language inputs. This research aims to understand if and how the languages included in an LLM's pre-training corpus affect its ability to handle code-mixed text, specifically for sentiment analysis.

Methods

The researchers evaluated several pre-trained language models (PLMs) on code-mixed sentiment analysis tasks. They employed two main experimental setups:

  1. In-language Finetuning: PLMs were fine-tuned on code-mixed datasets. This approach aims to see if the pre-training languages' influence is diminished when the model is directly trained on the target mixed-language data.
  2. Zero-Shot Setting: PLMs were fine-tuned on monolingual datasets (English and Hindi) and then tested on code-mixed data without any further code-mixed training. This setup is designed to reveal the impact of pre-training languages more starkly.

They also conducted experiments on Named Entity Recognition (NER), sarcasm detection, and Universal Dependency Parsing (UDPoS) to see if the findings for sentiment analysis generalized to other NLP tasks. Additionally, they used a language identification (LID) tool to analyze the composition of code-mixed sentences and how models performed on predominantly English versus predominantly Hindi parts of the text.

Datasets

The study utilized several publicly available code-mixed datasets, primarily scraped from social media platforms like Twitter and YouTube. These include:

  • Hinglish Datasets: IIITH-CodeMix and SAIL (Hindi-English)
  • Dravidian Datasets: TamilMixSentiment, MalayalamMixSentiment, and DravidianCodeMix (Tamil-English, Malayalam-English, Kannada-English)
  • Nigerian Pidgin (Naija) Datasets: NaijaVader and AfriSenti (English-Naija)

For the zero-shot experiments, they used monolingual English (SST-2) and Hindi (product reviews from Akhtar et al., 2016) sentiment analysis datasets for fine-tuning.

Key Insights and Ideas

  • Finetuning Masks Pre-training Influence: When PLMs are fine-tuned directly on code-mixed data, the specific languages they were pre-trained on seem to have a minimal impact on their performance. The fine-tuning process appears to adapt the models sufficiently, regardless of their pre-training background.
  • Zero-Shot Reveals Pre-training Power: In contrast, the zero-shot setting highlights the significant role of pre-training languages. Models pre-trained on languages relevant to the code-mixed data (e.g., Indic models for Indic languages) or code-mixed data itself (HingMBERT) show a marked advantage.
  • Language Composition Matters: The proportion of different languages within a code-mixed sentence or dataset can influence model performance. Models generally perform better on predominantly English parts of code-mixed text compared to predominantly Hindi parts.
  • Task Specificity: While sentiment analysis showed minimal differences in fine-tuning, other tasks like NER, sarcasm detection, and UDPoS also exhibited similar trends, suggesting a general pattern across different NLP tasks.

Key Results

  • Finetuning: For sentiment analysis, the accuracy scores across different PLMs (monolingual, multilingual, Indic, African, code-mixed) on code-mixed datasets showed very little variation, with differences often within 1-2 percentage points.
  • Zero-Shot: In the zero-shot setting, HingMBERT (pre-trained on Hinglish) significantly outperformed other models on Hinglish datasets, especially when fine-tuned on relevant monolingual data (Hindi or English). MuRIL showed strong performance with Hindi fine-tuning, while RoBERTa excelled with English fine-tuning.
  • Language ID Analysis: Models generally performed better at identifying and classifying English components of code-mixed text than Hindi components.

Limitations and Future Work

  • Ubiquity of English: A major limitation is that almost all PLMs included English in their pre-training data. This makes it difficult to disentangle the benefits of including relevant languages from the general benefit of including English.
  • Limited Non-English Monolingual Models: The study did not explore monolingual PLMs for languages other than English, partly due to the script differences (e.g., Devanagari for Indic languages) and the constraint of using Latin script for datasets.
  • Coarser-Grained LID: The lack of robust language identification tools for fine-grained intra-utterance analysis limits the depth of analysis, forcing a "mostly-English" vs. "mostly-Hindi" categorization.
  • Future Work: The authors suggest future work could examine code-mixed language pairs that do not involve English (e.g., Turkish-German) and explore other non-English monolingual PLMs. Further interpretability analysis is also recommended to understand why pre-training languages matter in some settings but not others.

Summary updated

BibTeX
@inproceedings{tatariya2023transfer,
  author = {Kushal Tatariya and Heather Lent and Miryam de Lhoneux},
  title = {{Transfer Learning for Code-Mixed Data: Do Pretraining Languages Matter?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure