TL;DR
This paper investigates how fine-tuning multilingual BERT (mBERT) with different types of code-mixed (CM) data impacts its performance on CM tasks. They found that real-world CM data yields the best performance improvements, and that any CM fine-tuning enhances the model's attention mechanisms' responsiveness to CM inputs.
What Problem Does This Paper Address?
Massively multilingual models like mBERT and XLMR have demonstrated impressive capabilities in handling tasks across many languages, even without explicit code-mixing (CM) training. However, CM, the phenomenon of mixing languages within a single sentence, is common in multilingual communities. While fine-tuning these models on CM data has shown promise, it's unclear which types of CM data are most beneficial and how this fine-tuning process actually changes the model's internal workings. This paper aims to demystify these effects by comparing different CM data types and analyzing the resulting model adaptations.
Methods
The researchers explored three types of code-mixed data for fine-tuning mBERT:
- Lexical Code-Mixing (l-CM): Simple substitution of words from one language with their counterparts from another within a sentence.
- Grammatical Code-Mixing (g-CM): Synthetically generated CM data that adheres to grammatical constraints of both languages, ensuring more natural sentence structures.
- Real Code-Mixing (r-CM): Naturally occurring CM data collected from real-world sources.
They fine-tuned mBERT on each of these data types for two language pairs (English-Spanish and English-Hindi) using a masked language modeling objective. To understand the impact, they performed two main analyses:
- Task-based Probing: Evaluating the fine-tuned models on 11 downstream CM tasks (including Language Identification, POS Tagging, NER, Sentiment Analysis, QA, and NLI).
- Structural Probing: Analyzing changes in the model's attention mechanisms using novel visualization techniques to understand how fine-tuning affects internal representations. This included examining inter-head distances and the "responsivity" of attention heads to CM stimuli.
Datasets
The paper utilized existing datasets for their experiments:
- Parallel Sentences: English-Spanish (enes) from Rijhwani et al. (2017) and English-Hindi (enhi) from Kunchukuttan et al. (2018) were used to generate synthetic CM data (l-CM and g-CM).
- Real-World CM Data (r-CM): Twitter data from Rijhwani et al. (2017) for enes, and data from online forums and Twitter by Chandu et al. (2018) and Patro et al. (2017) for enhi.
- Downstream Tasks: The researchers used tasks from the GLUECOS benchmark (Khanuja et al., 2020) and LINCE (Aguilar et al., 2020), specifically Language Identification (LID), Part-of-Speech (POS) Tagging, Named Entity Recognition (NER), Sentiment Analysis (SENT), Question Answering (QA), and Natural Language Inference (NLI).
Key Insights and Ideas
- Real-world data reigns supreme: Fine-tuning with naturally occurring code-mixed data (r-CM) consistently leads to the best performance improvements across various downstream tasks.
- Synthetic data's role: While synthetic CM data (l-CM, g-CM) alone can sometimes degrade performance compared to the base mBERT, it can be beneficial when used in a curriculum setting with r-CM, potentially alleviating data scarcity issues.
- Attention heads become CM-aware: Any form of CM fine-tuning, even with synthetic data, makes the model's attention heads more responsive to code-mixed inputs. This suggests a fundamental adaptation in how the model processes mixed-language text.
- Distributed representation: The study reinforces the idea that information for task-solving is distributed across multiple attention heads and layers, rather than being localized to specific components. This implies that models are robust to certain types of pruning.
- Linguistic complexity matters: The study hints that as the linguistic complexity of CM data increases (from lexical to grammatical to real-world), more attention heads become responsive to CM stimuli.
Key Results
- Performance Gains: Models fine-tuned with r-CM consistently outperformed models fine-tuned with l-CM or g-CM, and often surpassed the vanilla mBERT on tasks like SENT, NER, and LID (enes).
- Synthetic Data Drawback: Fine-tuning with only l-CM or g-CM data sometimes led to diminished performance compared to the stock mBERT model.
- Attention Responsivity: Structural probing revealed that CM fine-tuning increases the overall responsivity of attention heads to code-mixed stimuli. Specific heads were identified as being highly responsive to CM.
- Layer-wise Solvability: The study visualized how different layers contribute to task solvability, showing that simpler tasks are solved earlier in the network, while more complex tasks require deeper layers, a pattern consistent with BERT's "rediscovery" of the NLP pipeline.
Limitations and Future Work
- Data Scarcity: The paper acknowledges the ongoing challenge of obtaining large quantities of real-world CM data, which limits the scale of experiments and the potential for even greater performance gains.
- Curriculum Learning Nuances: While they experimented with curriculum learning for synthetic and real data, they couldn't derive meaningful insights, possibly due to the fixed 100k sentence limit.
- Analytical Nature: The current study is primarily analytical, observing how attention heads behave. Future work could expand this analysis to a wider range of domains and fine-tuning experiments to test the generalizability of their findings on distributed information representation.
- Responsivity Metric Improvement: The proposed "responsivity" metric for attention heads is a promising avenue, but the authors suggest it could be further improved to rigorously verify their observations.
Summary updated