TL;DR
Researchers from USC have developed a method to "audit" and improve emotion classification datasets by using Large Language Models (LLMs) to synthesize missing context. By generating 1-2 sentences of background information for ambiguous text snippets, they significantly improve the alignment between raw text and human-annotated labels, leading to better model generalization across different domains.
What Problem Does This Paper Address?
Text-based emotion classification is notoriously difficult because short snippets of text are often stripped of their original context. For example, the phrase "Wow!!!" could represent excitement, sarcasm, or even anger depending on the situation.
Existing datasets like GoEmotions contain thousands of such ambiguous samples. When human annotators are forced to pick a label for a three-word sentence, the resulting dataset becomes "noisy"—the labels might be technically correct in one interpretation but don't match the text's literal content. Re-annotating these massive datasets is too expensive and slow. The authors argue that instead of changing the labels to fit the text, we can use LLMs to expand the text to fit the labels.
Methods
The authors propose a two-step framework using GPT-3.5 and GPT-4:
- Context Evaluation: They prompt the LLM to act as a "Reddit user" and judge whether a specific emotion label is well-conveyed by the input text. If the LLM says "no," the sample is flagged as "Context-Absent."
- Context Generation: For these "Context-Absent" samples, the LLM is tasked with writing 1-2 additional sentences to append to the original post. Crucially, the prompt forbids the LLM from using the emotion word itself (e.g., it can't use the word "angry" to convey anger). This forces the model to create a situational narrative that naturally evokes the target emotion.
The goal is to create text that is faithful (consistent with the original snippet) and unambiguous (clearly pointing to the assigned label).
Datasets
- GoEmotions: The primary dataset used for auditing and training. It contains 58k Reddit comments labeled with 27 emotion categories.
- Evaluation Benchmarks (Out-of-Domain): To see if their "enhanced" training data actually helped, they tested on four external datasets:
- ISEAR: Self-reported emotional events.
- SemEval 2019 Task 3: Textual dialogues.
- Tweet Eval: Emotion recognition in tweets.
- Daily Dialog: Multi-turn conversations.
Key Insights and Ideas
- The "Teapot" Problem: The authors highlight a tension between faithfulness and unambiguity. If you add context that is too wild (e.g., turning a comment about booze into a story about being a "little teapot"), you lose the original meaning. Their prompting strategy specifically limits the LLM to two sentences to maintain a balance.
- Context as a Bridge: The most interesting finding is that adding context doesn't just make the training easier; it makes the resulting model more robust. By training on "unambiguous" versions of GoEmotions, the BERT-based models performed better on entirely different datasets (like tweets or dialogues), suggesting the model learned the semantics of the emotion rather than just memorizing noisy patterns.
- LLMs as Scalable Auditors: Instead of hiring thousands of humans to fix labels, we can use LLMs to "fix" the inputs. This shifts the bottleneck from human labor to prompt engineering.
Key Results
- Subjective Evaluation: Human workers on Amazon Mechanical Turk (MTurk) rated the "Context-Added" (GPT-4-CAM) samples significantly higher on a Likert scale for how well they expressed the target emotion compared to the original snippets.
- Objective Performance: Models trained on the context-enhanced datasets (GPT-X-CAM) showed consistent improvements in Macro F1 scores across almost all out-of-domain evaluation tasks compared to models trained on the original, ambiguous data.
- Sentiment Analysis: Interestingly, the method provided less benefit for simple sentiment analysis (positive/negative/neutral). The authors suggest this is because sentiment is a "coarser" task that doesn't require the fine-grained nuance that specific emotions (like "remorse" vs. "sadness") do.
Limitations and Future Work
- LLM Bias: The authors acknowledge that the synthesized context reflects the biases present in the LLM's training data, which could inadvertently "bake in" certain stereotypes about how emotions are expressed.
- Diversity of Expression: Analysis showed that LLMs tend to use a more limited vocabulary (e.g., using words like "truly" or "infuriating" frequently) compared to the "in-the-wild" diversity of original Reddit text.
- Future Directions: The team suggests expanding this to other modalities (like audio or video) and exploring "higher temperature" settings for data augmentation to create even more diverse training scenarios.
Summary updated