TL;DR
This paper introduces a new task: recognizing "embodied emotions" – physical sensations or movements linked to emotions – in natural language. They created a new dataset, CHEER, and found that existing emotion recognition models struggle with this task, motivating the development of new methods. Their work shows that weakly supervised learning can effectively train models for this task, even outperforming zero-shot approaches with large language models.
What Problem Does This Paper Address?
While we often think of emotions as purely mental states, they also manifest physically. These physical reactions, like a racing heart or clenched fists, are known as "embodied emotions." Recognizing these expressions in text is crucial for a deeper understanding of human emotions, potentially aiding in identifying behavioral traits and even problematic behaviors. However, existing natural language processing (NLP) models for emotion recognition often fail to capture these embodied expressions, as they tend to focus on explicit emotion words or general sentiment. This paper aims to bridge this gap by formalizing the task of recognizing embodied emotions in text and developing methods to address it.
Methods
The authors introduce a novel classification task: determining if a body part mention in a sentence is associated with an embodied emotion. They define embodied emotion as a physical movement or arousal evoked by emotion, with no purpose other than emotion expression.
To tackle this, they developed two primary methods for creating weakly labeled data:
- Dependency Pattern-Based Labeling: This method identifies body part words that are syntactically linked to explicit emotion words through "manner expressions." For example, in "He slammed his fist in anger," "slammed his fist" is identified as an embodied emotion linked to "anger." They use specific dependency relation patterns and an emotion lexicon to extract these instances.
- LLM Prompting: They leverage a large language model (GPT3.5) by creating a prompt template that includes instructions and input placeholders for body part mentions and sentences. The LLM then provides a yes/no answer to determine if the body part is involved in an embodied emotion.
For classification, they fine-tune a BERT-base model, referred to as the Embodied Emotion Classifier (EEC). They experiment with training this model on gold-labeled data, weakly labeled data from their two methods, and combinations thereof.
Datasets
The core contribution is the creation of the CHEER (Collection of Human annotations for Embodied Emotion Recognition) dataset.
- CHEER Dataset: This dataset contains 7,300 body part mentions extracted from personal blogs, with human annotations indicating whether the body part is involved in an embodied emotion or is neutral. It includes 56 distinct body parts and is publicly available.
- Weakly Labeled Data: The authors generated a large set of weakly labeled data using their pattern-based method (7,162 instances) and LLM prompting (7,000 Embodied Emotion and 56,648 Neutral instances).
Key Insights and Ideas
- Embodied Emotion is a Distinct Task: Existing emotion recognition models, even powerful ones like BERT-based classifiers, perform poorly on recognizing embodied emotions, highlighting its unique nature.
- Weak Supervision is Powerful: The paper demonstrates that weakly labeled data, generated automatically, can be highly effective for training embodied emotion classifiers. This is crucial given the cost and time involved in manual annotation.
- Complementarity of Weak Supervision Methods: Combining instances from both the pattern-based and LLM prompting methods for weak supervision yields better results than using either method alone, suggesting they capture different aspects of embodied emotion expressions.
- LLMs Struggle with Zero-Shot Embodied Emotion: While LLMs show impressive zero-shot capabilities on many tasks, they still struggle to accurately identify embodied emotions without specific fine-tuning.
Key Results
- Baseline Performance: Existing emotion classifiers achieved a maximum macro F1 score of only 58.2% on the CHEER dataset, with low recall for embodied emotions.
- Weak Supervision Performance:
- Training EEC with only weakly labeled data achieved a macro F1 score of 79.3% (using both pattern-based and LLM-prompted data).
- This is only 4.2 points lower than the model trained with gold data alone (macro F1 of 83.5%).
- Combined Supervision: Fine-tuning EEC with both gold and weakly labeled data further improved performance, achieving a macro F1 score of 85.4%, primarily driven by a significant increase in recall for embodied emotions.
- LLM Zero-Shot: GPT3.5 achieved the best zero-shot performance among LLMs, with a macro F1 of 70.2%, but its F1 score for the Embodied Emotion class was only 53.5%. Few-shot prompting with GPT3.5 surprisingly yielded worse results than zero-shot.
Limitations and Future Work
- Dataset Imbalance: The CHEER dataset has an unbalanced distribution of body parts, with the top 10 accounting for over 64% of instances. While they preserved the natural distribution, this "long tail" issue might affect performance on less frequent body parts.
- Text Corpus Bias: The dataset is derived from web blogs, which may bias it towards personal narratives and might not capture embodied emotions in other domains (e.g., medical contexts).
- Lack of Emotion Type Identification: The current work only identifies if an expression is an embodied emotion, not which specific emotion it represents.
- Future Work: The authors suggest extending the work to identify the specific type of emotion associated with embodied expressions and to explore more diverse text corpora. They also note that further data for low-frequency body parts could improve model performance.
Summary updated