TL;DR
This paper investigates how well different language models, from smaller multilingual ones to massive LLMs, can understand sentiment in languages other than English. They found that while smaller multilingual models excel at zero-shot learning, larger LLMs show better adaptability with a few examples, and proprietary LLMs like GPT-4 lead in zero-shot but are surpassed by fine-tuned public models in few-shot scenarios.
What Problem Does This Paper Address?
Sentiment analysis, a core NLP task, has traditionally focused on high-resource languages like English due to data scarcity in others. While machine translation and multilingual pre-trained models (like XLM-R, mT5) have helped bridge this gap, the cross-lingual capabilities of the newer, massive Large Language Models (LLMs) for sentiment analysis remain underexplored. This paper aims to fill this gap by systematically comparing the cross-lingual sentiment analysis performance of various public small multilingual language models (SMLMs) and English-centric LLMs, including proprietary ones, across different learning settings (zero-shot and few-shot).
Methods
The study employs two primary methodologies:
- Supervised Fine-Tuning (SFT):
- Classification-based SFT: Used for encoder-only models (e.g., mBERT, XLM-R), where a classification layer is added and fine-tuned.
- Instruction-based SFT: Used for encoder-decoder and decoder-only models (e.g., mT5, Llama-3), where models are prompted to generate sentiment class text based on instructions.
- In-Context Learning (ICL): This method is used for proprietary LLMs (GPT-3.5, GPT-4) due to data privacy constraints preventing fine-tuning. It involves providing examples within the prompt to guide the model's output.
Both SFT and ICL are applied in zero-shot (model trained/prompted on source language, tested on target) and few-shot (model trained/prompted on source language with N examples from the target language, tested on target) settings.
Datasets
The study utilizes proprietary sentiment datasets collected from internal call center transcripts for four languages: English, Spanish, French, and Chinese.
- English and Spanish: Manually annotated by a third-party vendor, ensuring high inter-annotator agreement.
- French and Chinese: Created using machine translation (GPT-4) from the annotated English data, followed by a review by native speakers to ensure quality.
The dataset sizes are:
- English: 30,000 instances for fine-tuning, 3,000 for development.
- Spanish: 600 instances for fine-tuning, 3,000 for testing.
- French: 600 instances for fine-tuning, 3,000 for testing.
- Chinese: 600 instances for fine-tuning, 3,000 for testing.
Key Insights and Ideas
- SMLMs punch above their weight in zero-shot: Smaller, multilingual models like XLM-R and mT5 demonstrate surprisingly strong zero-shot cross-lingual sentiment analysis performance, often outperforming much larger English-centric LLMs. This highlights their efficiency and effectiveness for low-resource scenarios.
- LLMs learn to adapt with few-shot data: While initially weaker in zero-shot, larger English-centric LLMs show remarkable improvement and adaptability when provided with even a small number of examples (few-shot learning) in the target language.
- Proprietary LLMs lead zero-shot, but public models catch up (and surpass) in few-shot: GPT-3.5 and GPT-4 excel in zero-shot cross-lingual sentiment analysis. However, when fine-tuned with few-shot data, several public LLMs can achieve comparable or even superior performance, suggesting that fine-tuning can unlock significant potential in open-source models.
- Conversational data nuances matter: The paper notes that sentiment expression can differ culturally and linguistically, with Spanish speakers sometimes describing situations rather than directly stating emotions, posing a challenge for sentiment analysis models.
Key Results
- Zero-shot:
- Proprietary models GPT-3.5 and GPT-4 showed the highest average F1 scores (79.4% and 78.1% respectively).
- Among public models, mT5-large (1.2B parameters) achieved the best zero-shot performance (avg. F1 of 73.5%), outperforming larger LLMs like Mistral-7B and Llama3-8B. XLM-R-large also performed well.
- Few-shot (60, 150, 600 shots):
- English-centric LLMs like Falcon-7B, Llama2-7B, and Llama3-8B showed significant improvements in few-shot settings, surpassing SMLMs.
- With 600-shot learning, mT5-xl (3.7B parameters) achieved performance comparable to larger LLMs.
- Proprietary LLMs (GPT-3.5, GPT-4) showed limited improvement in few-shot ICL compared to their zero-shot performance, with several fine-tuned public models surpassing them. For instance, in 600-shot, Llama3-8B achieved an average F1 of 84.6%, while GPT-4 was 78.0% (GPT-3.5 was not evaluated due to context limits).
Limitations and Future Work
- Limited scope of languages: The study focused on Spanish, French, and Chinese. It's unclear how models would perform in other low-resource languages with even less representation in pre-training data.
- Model architecture comparison: Due to the vast differences in model sizes and architectures (encoder-only, decoder-only, encoder-decoder), it's difficult to draw definitive conclusions about whether architectural choices play a role in cross-lingual sentiment analysis capabilities.
Future work could explore a wider range of languages and investigate the impact of specific architectural designs on cross-lingual transfer performance.
Summary updated