TL;DR
This paper presents the NYCU-NLP system for the EXALT 2024 shared task, which focuses on cross-lingual emotion and trigger detection in tweets. The team leveraged instruction-tuned Large Language Models (LLMs) and found that an ensemble of Starling (7B) and Llama 3 (8B) models achieved competitive results, ranking sixth for emotion detection and fifth for trigger detection among participating systems.
What Problem Does This Paper Address?
The paper tackles the challenge of understanding emotions expressed in text across different languages, specifically within the context of social media (tweets). While emotion detection is a well-studied area, the EXALT task emphasizes the explainability of cross-lingual emotion detection. This means not only identifying the emotion but also pinpointing the specific words or emoticons that convey that emotion (trigger detection). The task also introduces a cross-lingual aspect, requiring models to generalize from English training data to predict emotions in Dutch, Russian, Spanish, French, and English evaluation data. This presents a significant hurdle due to linguistic diversity and potential cultural nuances in emotional expression.
Methods
The NYCU-NLP system's core methodology involves two main stages:
- Instruction Fine-tuning of LLMs: The researchers selected three publicly available instruction-tuned LLMs: Mistral (7B), Starling (7B), and Llama 3 (8B). These models were further fine-tuned on the task-specific training data using custom prompts designed for both emotion classification and trigger identification. The prompts were crafted to guide the LLMs to act as sentiment analysis masters, classifying emotions into six categories or identifying trigger words/emoticons using a specific "#" notation.
- Assembly Mechanism: For inference, the system employed a voting-based assembly mechanism. Each fine-tuned LLM independently processed the test instances. The final prediction was determined by a majority vote among the models. For emotion detection, if no clear majority emerged, the "neutral" category was used as a fallback. For trigger detection, any word or emoticon identified as a trigger by at least one model was considered a system output.
The paper also mentions the use of Low-Rank Adaptation (LoRA) for efficient instruction-tuning, which injects trainable matrices into the transformer architecture while keeping the pre-trained weights frozen.
Datasets
The datasets were provided by the task organizers for the EXALT shared task.
- Emotion Detection Subtask:
- Training Data: 5000 instances (English)
- Development Data: 3000 instances
- Test Data: 2500 instances
- Binary Trigger Detection Subtask:
- Training Data: 3000 instances
- Development Data: 300 instances
- Test Data: 832 instances
The paper notes that no data augmentation was used.
Key Insights and Ideas
- LLM Ensembles Shine: The study reinforces the common finding that ensemble methods, combining the predictions of multiple models, often outperform individual models. This was particularly evident in their results, where the assembly of Starling and Llama 3 yielded the best performance.
- Instruction Tuning is Key: The success of the system hinges on effectively adapting pre-trained LLMs to the specific tasks through instruction fine-tuning. The carefully designed prompts played a crucial role in guiding the LLMs' behavior.
- Cross-Lingual Transfer is Possible (with LLMs): The fact that LLMs, pre-trained on multilingual data, could be fine-tuned to perform reasonably well on cross-lingual tasks without explicit machine translation suggests their inherent ability to capture and transfer linguistic and emotional nuances across languages.
- Trigger Detection is Challenging: While emotion detection saw competitive results, trigger detection, which requires pinpointing specific words, proved to be a more difficult subtask, as indicated by the lower rankings.
Key Results
- Best Performing Model: The assembly of Starling (7B) and Llama 3 (8B) was the best-performing submission.
- Emotion Detection: This ensemble achieved an F1 score of 0.5951 on the development set, ranking sixth out of 17 participating systems. On the test set, the F1 score was 0.5961.
- Binary Trigger Detection: The same ensemble achieved a token F1 score of 0.5636 on the development set, ranking fifth out of 7 participating systems. On the test set, the token F1 score was 0.5636.
- Individual Model Performance: Llama 3 (8B) individually outperformed Mistral (7B) and Starling (7B) on both subtasks in terms of F1 scores on the test set.
Limitations and Future Work
- Limited Prompt Engineering: The authors acknowledge that due to computational resource constraints, they did not explore advanced prompt engineering techniques. They believe further improvements could be made by optimizing these prompts.
- No Machine Translation: The system did not employ machine translation, relying solely on the multilingual capabilities of the pre-trained LLMs. While this worked, future work could explore how integrating translation might further enhance performance.
- No Novel Model Architecture: The paper states that they did not propose a new model architecture for the shared task, focusing instead on leveraging and assembling existing LLMs.
- Future Work: The authors express interest in exploring other advanced LLMs and further improving performance in future work, suggesting this pilot study is a foundational step. They also mention the need for more advanced explorations to enhance system performance.
Summary updated