TL;DR
This paper explores using large language models (LLMs) like ChatGPT for emotion detection in code-switched SMS messages (Roman Urdu and English). Despite the challenges of colloquial language and low-resource settings, their few-shot prompting approach with LLMs outperformed traditional ML models and even a BERT-based system, achieving a macro F1 score of 0.7038.
What Problem Does This Paper Address?
Emotion detection in text is inherently tricky due to the subjective nature of emotions, their dependence on context, and the varied ways they are expressed. These challenges are amplified when dealing with code-switched text, where multiple languages are mixed within a single message, especially in informal settings like SMS. Furthermore, high-quality labeled data for such specific scenarios is scarce, and low-resource languages like Urdu present additional hurdles. This paper tackles the problem of accurately detecting 11 different emotions (plus a neutral class) in code-mixed Roman Urdu and English SMS messages, a task that is difficult for both humans and traditional ML models.
Methods
The authors leveraged generative pretrained transformers (GPTs), specifically ChatGPT, for emotion detection. Their approach involved prompt engineering and zero-shot/few-shot learning. Initially, they experimented with zero-shot classification, where ChatGPT was asked to translate and then classify, or directly classify. This proved challenging, with LLM responses sometimes being difficult to parse and accuracy being low.
The most successful method involved a few-shot learning approach. For each message in the development or test set, they identified the 100 most similar messages from the training set using cosine similarity on OpenAI embeddings. These similar examples were then used as few-shot examples to guide ChatGPT in classifying the target message. This iterative prompt engineering and few-shot learning strategy was key to their success.
Datasets
The dataset used for this task was originally proposed by Ameer et al. (2022). It consists of code-mixed Roman Urdu and English SMS messages, annotated for 12 emotions (anger, anticipation, disgust, fear, joy, love, optimism, pessimism, sadness, surprise, trust, and neutral). The original training set had approximately 9.5k examples, with around 1190 examples each in the development and test sets. The authors performed deduplication and removed overlapping data points between sets, resulting in a training set of about 6.1k examples.
Key Insights and Ideas
- LLMs as powerful few-shot learners for code-switching: The paper demonstrates that LLMs, when guided by carefully crafted few-shot examples, can effectively handle the complexities of code-switched text for emotion detection, even outperforming specialized models.
- Prompt engineering is crucial but can be iterative: The quality of the LLM's output is highly dependent on the prompt. The authors found that direct classification prompts, especially when combined with relevant few-shot examples, were more effective than translation-first approaches.
- LLMs can exhibit "excessive perfectionism": The authors observed that ChatGPT sometimes refused to perform tasks or complained about the input quality when asked to be highly accurate, suggesting a need for careful prompt design to avoid triggering such responses.
- Code-switching and low-resource languages are significant challenges: The paper highlights that even for advanced LLMs, dealing with colloquial code-switched text and languages with less training data (like Urdu) remains a difficult task.
Key Results
The authors' best performing approach, using few-shot learning with ChatGPT, achieved a macro F1 score of 0.7038 and an accuracy of 0.7313 on the blind test set. This performance surpassed their baseline XGBClassifier (macro F1: 0.68) and the baseline BERT-base-multilingual-cased system (macro F1: 0.7014), securing them 4th place in the shared task.
Limitations and Future Work
The authors acknowledge several limitations:
- Prompt dependency: The results are highly sensitive to prompt design, requiring significant effort to find optimal prompts for specific use cases.
- Output volatility: While generally stable, ChatGPT can sometimes produce extraneous text or categories outside the predefined list, requiring post-processing.
- Limited context window: ChatGPT's context window (4096 tokens) makes processing large datasets challenging, necessitating splitting them into smaller pieces.
- API costs: Using the ChatGPT API incurs costs, which can be a consideration for extensive prompt engineering or large-scale few-shot learning.
For future work, they anticipate that GPT-4 will likely yield even better results, noting that its API was not available during their experiments. They also suggest that further exploration of prompt engineering techniques and potentially larger context window models could improve performance.
Summary updated