Towards Fine-grained Classification of Climate Change related Social Media Text

Roopal Vaid, Kartikey Pant, Manish Shrivastava

Abstract

Traditional sentiment analysis often fails to capture the complexity of climate change discourse on social media. This paper introduces three new datasets—ClimateStance, ClimateEng, and ClimateReddit—to enable more nuanced analysis. By leveraging Transformer-based models like RoBERTa, the researchers demonstrate high accuracy in categorizing text into specific themes and detecting stances on climate action.

Introduction

Climate change is a deeply polarizing topic, yet understanding public opinion remains a challenge. While polling provides snapshots, social media offers a continuous stream of the "vox populi." However, existing research often relies on binary sentiment (pro vs. anti), which lacks the detail necessary for effective policy-making or targeted communication. This study addresses the gap in fine-grained classification and stance detection, moving beyond simple sentiment to identify specific topics and nuanced positions.

Datasets and Task Definition

The researchers developed three novel benchmarks to facilitate this research:

  1. ClimateStance: 3,777 tweets from COP 25, labeled for stance: Favor, Against, or Ambiguous.
  2. ClimateEng: The same tweets categorized into five fine-grained themes: Disaster, Ocean/Water, Agriculture/Forestry, Politics, and General.
  3. ClimateReddit: 6,591 comments from climate subreddits, primarily used for semi-supervised learning.
Examples of fine-grained classification categories for climate-related text.

Methodology

The study employed a multi-stage approach. First, supervised learning was used to benchmark Transformer models (BERT, RoBERTa, DistilBERT) against a FastText baseline. Second, a semi-supervised "teacher-student" method utilized labeled Twitter data to pseudo-label a larger Reddit dataset. Finally, linguistic analysis using SpaCy (POS tagging and NER) explored the stylistic differences between stances.

Experimental Results

The results highlight the varying difficulty of these NLP tasks. Fine-grained classification proved highly successful, with RoBERTa-Large achieving an F1 score of 0.735 and an accuracy of 83.07% on the ClimateEng dataset.

Performance metrics for various models on the fine-grained classification task (ClimateEng).

Stance detection, however, was more challenging. RoBERTa-Base performed best with an F1 of 0.510. The researchers noted that sarcasm and ambiguous intent often confused both human annotators and models, leading to lower performance in stance detection compared to topic classification.

Discussion: Semi-Supervised Learning and Linguistic Patterns

The inclusion of pseudo-labeled Reddit data yielded mixed results. While it significantly boosted fine-grained classification (F1 increased from 0.775 to 0.834), it degraded stance detection performance. This was attributed to a severe class imbalance in the Reddit data, where the "Favor" stance was overwhelmingly dominant, causing the model to lose its ability to distinguish minority stances.

Linguistic analysis revealed distinct "fingerprints" for different stances. "Favor" tweets frequently used proper nouns and nouns, indicating a focus on specific policies and entities. In contrast, "Against" tweets relied more on adjectives and adverbs, often employing emotional or sarcastic language to dismiss climate concerns.

Conclusion

This work demonstrates that while topic classification is relatively robust, detecting nuanced stances remains a complex challenge due to sarcasm and class imbalance. Future research should explore multi-modal and multilingual data to further refine our understanding of global climate discourse.

Summary updated

Abstract

With climate change becoming a cause of concern worldwide, it becomes essential to gauge people’s reactions. This can help educate and spread awareness about it and help leaders improve decision-making. This work explores the fine-grained classification and Stance detection of climate change-related social media text. Firstly, we create two datasets, ClimateStance and ClimateEng, consisting of 3777 tweets each, posted during the 2019 United Nations Framework Convention on Climate Change and comprehensively outline the dataset collection, annotation methodology, and dataset composition. Secondly, we propose the task of Climate Change stance detection based on our proposed ClimateStance dataset. Thirdly, we propose a fine-grained classification based on the ClimateEng dataset, classifying social media text into five categories: Disaster, Ocean/Water, Agriculture/Forestry, Politics, and General. We benchmark both the datasets for climate change stance detection and fine-grained classification using state-of-the-art methods in text classification. We also create a Reddit-based dataset for both the tasks, ClimateReddit, consisting of 6262 pseudo-labeled comments along with 329 manually annotated comments for the label. We then perform semi-supervised experiments for both the tasks and benchmark their results using the best-performing model for the supervised experiments. Lastly, we provide insights into the ClimateStance and ClimateReddit using part-of-speech tagging and named-entity recognition.
BibTeX
@inproceedings{vaid2022towards,
  author = {Roopal Vaid and Kartikey Pant and Manish Shrivastava},
  title = {{Towards Fine-grained Classification of Climate Change related Social Media Text}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-srw.35},
}
Figure