TEII: Think, Explain, Interact and Iterate with Large Language Models to Solve Cross-lingual Emotion Detection

Long Cheng, Qihao Shao, Christine Zhao, Sheng Bi, Gina-Anne LevowACL 2024

TL;DR

This paper tackles the challenge of cross-lingual emotion detection in social media by leveraging Large Language Models (LLMs). They introduce an "Agentic Workflow" that uses multiple LLMs to collaborate and refine predictions, achieving a strong F1-score of 0.6046 and ranking second in the EXALT shared task. This approach not only improves accuracy but also offers explainability for the detected emotions.

What Problem Does This Paper Address?

Understanding public opinion and global trends requires analyzing emotions expressed on social media across different languages. However, cross-lingual emotion detection is a complex task due to linguistic diversity and cultural nuances in expressing emotions. Traditional methods often struggle with this complexity. This paper aims to improve the accuracy and explainability of emotion detection in multilingual tweets by exploring the capabilities of advanced LLMs and novel multi-agent interaction strategies.

Methods

The researchers experimented with several approaches:

  • LLM-based methods:
    • Fine-tuning GPT3.5: Training a specific LLM on the task data.
  • Zero-shot learning (GPT-4, Claude3): Using LLMs without task-specific training, relying on prompts to guide them.
  • Few-shot learning: Providing a few examples to the LLM to help it understand the task.
  • Zero-Shot with Explanation (ZSE) and Correction (ZSEC): Prompting LLMs to not only predict the emotion but also provide an explanation for their choice. ZSEC further refines this by using a second LLM to review and potentially correct the initial prediction and explanation.
  • Non-LLM-based techniques:
    • Embedding-based BiLSTM and KNN: Utilizing classical machine learning models with multilingual embeddings.
  • Novel Agentic Workflow (AWF) approaches:
    • Multi-Iteration Agentic Workflow (MIAWF): This method involves multiple LLM "agents" collaborating. The top-performing models are identified, and then another LLM agent is tasked with adjudicating between their outputs, potentially iterating to improve the decision.
  • Multi-Binary-Classifier Agentic Workflow (MBCAWF): This approach hypothesizes that binary classification is easier for LLMs. It breaks down the multi-class emotion detection into multiple binary classification tasks (e.g., "Is this tweet angry?"). A final LLM then synthesizes these binary decisions, with a special check for "Neutral" emotions.
  • Ensembling: Combining the predictions of multiple models (including individual LLMs, Agentic Workflows, and classical models) using unweighted voting to achieve a more robust final prediction.

Datasets

The primary dataset used is the EXALT shared task dataset for cross-lingual emotion detection in tweets. This dataset contains tweets in multiple languages, and the task is to classify them into six emotion categories: Love, Joy, Anger, Fear, Sadness, and Neutral. The paper mentions using this dataset for both training and evaluation.

Key Insights and Ideas

  • LLMs excel at multilingual emotion detection: The study confirms that LLM-based approaches, particularly when prompted effectively, can achieve strong performance on this task.
  • Explanation enhances performance: Prompting LLMs to provide explanations (ZSE) not only aids human understanding but also seems to improve the accuracy of the emotion prediction itself.
  • Agentic Workflows offer a powerful synergy: The Agentic Workflow approach, by enabling LLMs to interact and refine each other's outputs, significantly boosts performance beyond individual LLMs. This highlights the benefit of collaborative AI.
  • Ensembling is key to top performance: Combining diverse models, including LLMs and Agentic Workflows, through ensembling yielded the best overall results, demonstrating that a single model is rarely optimal.
  • Subjectivity in emotion is a challenge: Manual error analysis revealed that even humans can disagree on emotion labels, and LLM explanations, while sometimes incorrect according to the gold standard, can still be reasonable.

Key Results

The proposed system, Ensemble-19, achieved an F1-score of 0.6046 on the test dataset, outperforming the EXALT baseline by approximately 0.17 F1-score and ranking second among competing systems.

Individually, the Zero-Shot with Explanation and Correction (ZSEC-gpt4o) model performed strongly with an F1-score of 0.5726. The Multi-Binary-Classifier Agentic Workflow (MBCAWF) and Multi-Iteration Agentic Workflow (MIAWF) models also showed competitive results, exceeding 0.55 F1-score.

The paper also notes that per-emotion performance varied, with "Anger" and "Neutral" scoring highest (around 0.73 and 0.72 F1-score respectively) and "Love" and "Fear" scoring lowest (around 0.47 and 0.53 F1-score).

Limitations and Future Work

  • Closed-source LLMs: The reliance on proprietary LLMs (OpenAI, Anthropic) makes exact reproduction challenging due to their evolving nature and potential for non-deterministic outputs.
  • Nonsensical outputs: LLMs occasionally produced nonsensical outputs that required manual correction, impacting efficiency.
  • Cost and latency: LLM inference can be computationally expensive and time-consuming, especially with longer explanations.
  • Limited language scope: The study focused on high-resource languages, and its applicability to low-resource languages remains uncertain.

Future work could involve applying the Agentic Workflow methodology to a broader range of sentiment analysis tasks and exploring its utility in other NLP domains. Further investigation into the performance on low-resource languages is also suggested.

Summary updated

BibTeX
@inproceedings{cheng2024teii,
  author = {Long Cheng and Qihao Shao and Christine Zhao and Sheng Bi and Gina-Anne Levow},
  title = {{TEII: Think, Explain, Interact and Iterate with Large Language Models to Solve Cross-lingual Emotion Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure