TL;DR
This paper explores the challenges of annotating emotional language in environmental communication, revealing significant disagreement among human annotators. By analyzing these disagreements, the authors offer insights into how individual perspectives shape interpretation and propose methods to improve future annotation efforts in NLP and environmental research.
What Problem Does This Paper Address?
Environmental crises demand coordinated global action, and emotional language plays a crucial role in motivating this. However, understanding how this emotional language is used and interpreted, especially by activist groups, is complex. While NLP has made strides in sentiment analysis, the nuanced and subjective nature of emotional language in environmental discourse presents a significant challenge. Existing datasets often focus on general sentiment or denial, leaving a gap in understanding the specific emotional expressions used by activist groups and the inherent subjectivity in annotating them. This paper tackles the problem of annotator disagreement in this domain, highlighting the need for more human-centered approaches to NLP.
Methods
The authors conducted an annotation study using a multi-label approach, where each tweet could be assigned multiple emotion and sentiment categories. Three experienced annotators were tasked with labeling tweets from the Extinction Rebellion (XR) group. To understand the sources of disagreement, they employed several analytical methods:
- Quantitative Analysis: This included examining the absolute frequency distribution of labels per annotator, calculating inter-annotator agreement (IAA) using Fleiss' Kappa and Cohen's Kappa, and performing Pointwise Mutual Information (PMI) analysis to identify potential lexical biases.
- Clustering-Based Topic Modeling: BERTopic was used to identify underlying themes in the tweets and analyze how annotators' emotion labels correlated with these topics.
- Temporal Analysis: The study tracked annotation patterns over time to see if annotators' labeling behavior changed as they progressed through the dataset.
- Qualitative Interviews: In-depth interviews with the annotators were conducted to gather insights into their individual perspectives, emotional responses to environmental issues, and mental associations with environmental groups.
Datasets
The study utilized a newly created dataset consisting of 2,199 English-language tweets from the environmental activist group Extinction Rebellion (XR), extracted between September 2024 and published between 2022 and 2024. This dataset, including the annotations, is made publicly available.
Key Insights and Ideas
- Subjectivity is Inevitable: The study strongly suggests that annotating emotional language, especially in charged contexts like environmental activism, is inherently subjective. There isn't a single "ground truth," and annotator disagreement is a valuable signal, not just noise.
- Individual Perspectives Matter: Annotators' personal emotional responses to environmental crises, their background knowledge, and even their mental imagery associated with activist groups significantly influence their labeling.
- Lexical Biases Exist: Annotators can develop tendencies to associate specific words with certain emotions, sometimes based on explicit terms and other times on more contextual or indirect associations.
- Topic Influence: The topic of a tweet can influence annotator agreement, with some topics leading to more consistent labeling than others.
- Temporal Shifts: Annotators' labeling patterns can evolve over time as they become more familiar with the task and data.
Key Results
- Substantial Disagreement: The study found significant variation in annotation choices across the three annotators, with Fleiss' Kappa values ranging from moderate to slight agreement depending on the category.
- Varying Labeling Tendencies: One annotator (A2) was more conservative in assigning labels, while another (A1) assigned the highest frequency of labels.
- Specific Emotion Challenges: Categories like PRIDE and JOY were least frequently assigned, while ANGER and CONCERN showed more complex patterns of agreement and disagreement.
- Qualitative Insights: Interviews revealed that annotators' personal emotional responses (e.g., concern, anger, fear) and their perceived affectedness by environmental issues correlated with their labeling behavior.
Limitations and Future Work
The authors acknowledge several limitations:
- Small, Homogeneous Annotator Pool: The study involved only three annotators, all female students from Germany. This limits the generalizability of the findings. Future work should recruit a more diverse group of annotators.
- Single Organization Focus: The dataset solely comprises tweets from Extinction Rebellion. Expanding to include a wider range of environmental groups would provide a more robust understanding of emotional language in the field.
- Qualitative Interview Limitations: While interviews provided valuable insights, they were conducted after the annotation process, potentially influencing recall. Real-time self-assessments during annotation could offer more precise data.
Future work should focus on:
- Developing Perspectivist Frameworks: Adopting annotation strategies that explicitly account for diverse perspectives, such as weak perspectivism or citizen science approaches.
- Collecting Annotator Metadata: Gathering sociodemographic information, domain expertise, and personal stances from annotators before annotation to better understand their biases.
- Improving Annotation Methods: Exploring LLMs to complement human labeling and inferring writer intentions.
- Expanding Datasets: Including a broader range of environmental groups and analyzing temporal aspects of annotation subjectivity.
Summary updated