TL;DR
This paper tackles the bottleneck of manual data annotation for emotion intensity prediction by automating the process using large language models (LLMs). They found that Best-Worst Scaling (BWS), a comparative annotation method, yields the most reliable automated annotations, outperforming direct rating scales. Furthermore, LLM-generated annotations, particularly when using BWS, enable training a regression model that performs on par with models trained on human annotations.
What Problem Does This Paper Address?
Creating large, high-quality datasets for machine learning tasks is a significant hurdle, especially for nuanced tasks like predicting emotion intensity. Manual annotation is time-consuming and expensive. While automated annotation methods exist for categorical tasks, continuous value prediction (like intensity) has been less explored. Humans themselves struggle with consistent continuous value assignments, often performing better with comparative tasks like Best-Worst Scaling (BWS). This paper investigates whether LLMs, when applied to annotation tasks, exhibit similar patterns, and if BWS can be effectively automated to produce reliable data for emotion intensity modeling.
Methods
The authors propose a workflow to automate the creation of annotation datasets for emotion intensity. This involves:
- Tuple Creation: Grouping texts into sets (tuples) for annotation. For BWS, they use 4-text tuples, a common practice.
- Prompting: Crafting specific prompts for LLMs (GPT-3.5-turbo, GPT-3, Llama2) to perform different annotation tasks:
- Rating Scales (RS): Direct numerical assignment.
- Rating Scales Tuples (RS-T): Rating texts within a tuple.
- Paired Comparisons (PC): Comparing pairs of texts.
- Best-Worst Scaling (BWS): Identifying the most and least intense texts within a tuple.
- Score Calculation: For RS and RS-T, the LLM's output is used directly. For PC and BWS, a counting method is applied, and results are normalized.
- Regressor Training: A transformer-based regressor (RoBERTa) is trained on the LLM-generated annotations.
They compare the LLM-generated annotations directly against human annotations and indirectly by evaluating the performance of the trained regressor.
Datasets
- Affect-in-Tweets Dataset (AIT): Used for training and evaluation. This dataset was originally annotated using BWS for emotion intensity (joy, sadness, anger, fear). The paper uses the AIT dataset's original human annotations as a benchmark.
- SemEval-2007 Task 14: Affective Text dataset: Used for a second validation of their method. This dataset consists of news headlines annotated for six emotions using rating scales.
Key Insights and Ideas
- BWS Shines for LLM Annotations: Just as BWS is more reliable for human annotators on continuous tasks, it also proves to be the most reliable method for LLM-based emotion intensity annotation.
- LLMs Mimic Human Annotation Challenges: LLMs, like humans, perform better with comparative tasks (BWS) than direct rating scales for continuous values.
- Automated Annotations Can Be "Good Enough": LLM-generated annotations, particularly from BWS, allow for training a regression model that performs comparably to models trained on original human annotations. This significantly reduces the need for manual labeling.
- Prompt Engineering Matters: The way prompts are structured, especially the level of detail in scale descriptions, significantly impacts LLM performance. More descriptive prompts lead to better results.
- Tuple Count is Key for Quality: Increasing the number of tuples used for annotation (even with a single LLM pass per tuple) leads to improved annotation quality and downstream model performance.
Key Results
- Direct Comparison: BWS consistently outperformed RS and RS-T across all emotions in direct comparisons with human annotations. Paired Comparisons (PC) also performed well but were significantly less efficient.
- Indirect Comparison (Regressor Performance): Models trained on LLM-generated BWS annotations achieved performance on par with models trained on original human annotations. For example, with 2N tuples, the average performance was 73.5, compared to 78.3 for humans. Increasing tuple counts to 12N improved performance to 74.8.
- LLM Performance: GPT-3.5 significantly outperformed GPT-3 and Llama2 in annotation quality.
- Cost-Effectiveness: While BWS requires more annotations than basic rating scales (2N vs. N), the substantial increase in performance is deemed worth the cost. PC was found to be too inefficient.
Limitations and Future Work
- Black-Box Nature of LLMs: The reliance on proprietary LLMs like GPT-3.5 makes the system less transparent and potentially less reliable if APIs change. Future work should explore open-source generative models.
- Token Limits: Current LLM token limits can be a challenge for annotating very long texts.
- Task Specificity: For highly specialized tasks, LLMs might require fine-tuning, which could negate the benefit of avoiding manual annotation.
- Generalizability: The study focused on emotion intensity. Further research is needed to validate these findings across a wider range of NLP tasks with continuous labels.
- Bias: The potential for LLMs to introduce different biases than human annotators needs further investigation.
- Human vs. LLM Differences: While LLM annotations can be comparable, there are subtle differences. LLMs tend to be more consistent with explicitly mentioned emotions but may interpret implicit information less carefully than humans.
The authors suggest exploring other datasets and tasks, such as toxicity or word similarity, and investigating the impact of LLM architecture and training on annotation quality.
Summary updated