Abstract
The lack of contextual information in text data can make the annotation process of text-based emotion classification datasets challenging. As a result, such datasets often contain labels that fail to consider all the relevant emotions in the vocabulary. This misalignment between text inputs and labels can degrade the performance of machine learning models trained on top of them. As re-annotating entire datasets is a costly and time-consuming task that cannot be done at scale, we propose to use the expressive capabilities of large language models to synthesize additional context for input text to increase its alignment with the annotated emotional labels. In this work, we propose a formal definition of textual context to motivate a prompting strategy to enhance such contextual information. We provide both human and empirical evaluation to demonstrate the efficacy of the enhanced context. Our method improves alignment between inputs and their human-annotated labels from both an empirical and human-evaluated standpoint.
Introduction
Text-based emotion classification is a fundamental task in natural language processing, yet it is notoriously difficult because short snippets of text are often stripped of their original context. For example, the phrase "Wow!!!" could represent excitement, sarcasm, or even anger depending on the situation. When human annotators are forced to assign labels to such ambiguous samples, the resulting datasets—such as the widely used GoEmotions—become "noisy." The labels may be technically correct under one interpretation, but they often fail to align with the literal content of the text snippet.
Traditionally, the solution to noisy data is re-annotation. However, re-annotating massive datasets is economically prohibitive and slow. Researchers from USC propose a paradigm shift: instead of changing the labels to fit the text, we can use Large Language Models (LLMs) to expand the text to fit the labels. By synthesizing missing context, we can "audit" and improve the alignment of existing datasets, leading to more robust machine learning models.
The Context-Added Method (CAM)
The researchers developed a two-step framework utilizing GPT-3.5 and GPT-4 to enhance dataset quality:
- Context Evaluation (Auditing): The LLM is prompted to act as a "Reddit user" and judge whether a specific human-annotated emotion label is well-conveyed by the input text. If the LLM determines the emotion is not clearly present, the sample is flagged as "Context-Absent."
- Context Generation (Synthesis): For these flagged samples, the LLM is tasked with writing 1-2 additional sentences of background information to append to the original post. A critical constraint is applied: the prompt forbids the LLM from using the emotion word itself (e.g., it cannot use the word "angry" to convey anger). This forces the model to create a situational narrative that naturally evokes the target emotion, ensuring the text remains nuanced.
The goal of this framework is to create text that is both faithful (consistent with the original snippet) and unambiguous (clearly pointing to the assigned label).
Experimental Setup
The primary dataset for auditing and training was GoEmotions, which contains 58,000 Reddit comments labeled with 27 emotion categories. To evaluate the effectiveness of the context-enhanced data, the researchers tested the resulting models on four external, out-of-domain (OOD) benchmarks:
- ISEAR: Self-reported emotional events.
- SemEval 2019 Task 3: Textual dialogues.
- Tweet Eval: Emotion recognition in tweets.
- Daily Dialog: Multi-turn conversations.
By testing on these diverse datasets, the team could determine if the LLM-generated context helped the model learn generalizable emotional semantics rather than just memorizing Reddit-specific patterns.
Key Insights
The Faithfulness-Unambiguity Trade-off
The authors highlight a tension they call the "Teapot" problem. If an LLM adds context that is too imaginative (e.g., turning a simple comment about a beverage into a surreal story about being a "little teapot"), it loses the original meaning. By limiting the LLM to two sentences, the CAM framework maintains a balance between providing enough information to justify the label and staying true to the original user's intent.
Context as a Bridge for Generalization
One of the most significant findings is that adding context does more than just simplify the training set; it makes the resulting model more robust. BERT-based models trained on the "unambiguous" versions of GoEmotions performed better on entirely different datasets, such as tweets and dialogues. This suggests that the enhanced context allows the model to learn the underlying semantics of emotions more effectively than noisy, short-form text.
LLMs as Scalable Auditors
This method shifts the bottleneck of dataset improvement from human labor to prompt engineering. Using LLMs to "fix" inputs rather than hiring thousands of humans to fix labels represents a scalable way to improve legacy datasets that were previously considered too noisy for high-stakes applications.
Results and Performance
- Subjective Human Evaluation: Human workers on Amazon Mechanical Turk (MTurk) were asked to rate how well the text expressed the target emotion. The "Context-Added" samples (GPT-4-CAM) received significantly higher Likert scale ratings compared to the original, ambiguous snippets.
- Objective Metrics: Models trained on context-enhanced datasets showed consistent improvements in Macro F1 scores across nearly all out-of-domain tasks.
- Sentiment vs. Emotion: Interestingly, the method provided less benefit for simple sentiment analysis (positive/negative/neutral). The authors suggest that sentiment is a "coarser" task; it does not require the fine-grained situational nuance that specific emotions like "remorse" or "grief" demand.
Limitations and Future Work
While effective, the authors acknowledge that LLM-synthesized context can reflect biases present in the model's training data, potentially baking in stereotypes about emotional expression. Furthermore, LLMs tend to use a more formal or limited vocabulary (e.g., overusing words like "truly") compared to the raw diversity of human speech on Reddit. Future research will explore expanding this method to multimodal data (audio and video) and using higher "temperature" settings to increase the diversity of the synthesized narratives.
Summary updated