Abstract
Traditional sentiment analysis often fails to capture the complexity of climate change discourse on social media. This paper introduces three new datasets—ClimateStance, ClimateEng, and ClimateReddit—to enable more nuanced analysis. By leveraging Transformer-based models like RoBERTa, the researchers demonstrate high accuracy in categorizing text into specific themes and detecting stances on climate action.
Introduction
Climate change is a deeply polarizing topic, yet understanding public opinion remains a challenge. While polling provides snapshots, social media offers a continuous stream of the "vox populi." However, existing research often relies on binary sentiment (pro vs. anti), which lacks the detail necessary for effective policy-making or targeted communication. This study addresses the gap in fine-grained classification and stance detection, moving beyond simple sentiment to identify specific topics and nuanced positions.
Datasets and Task Definition
The researchers developed three novel benchmarks to facilitate this research:
- ClimateStance: 3,777 tweets from COP 25, labeled for stance: Favor, Against, or Ambiguous.
- ClimateEng: The same tweets categorized into five fine-grained themes: Disaster, Ocean/Water, Agriculture/Forestry, Politics, and General.
- ClimateReddit: 6,591 comments from climate subreddits, primarily used for semi-supervised learning.
Methodology
The study employed a multi-stage approach. First, supervised learning was used to benchmark Transformer models (BERT, RoBERTa, DistilBERT) against a FastText baseline. Second, a semi-supervised "teacher-student" method utilized labeled Twitter data to pseudo-label a larger Reddit dataset. Finally, linguistic analysis using SpaCy (POS tagging and NER) explored the stylistic differences between stances.
Experimental Results
The results highlight the varying difficulty of these NLP tasks. Fine-grained classification proved highly successful, with RoBERTa-Large achieving an F1 score of 0.735 and an accuracy of 83.07% on the ClimateEng dataset.
Stance detection, however, was more challenging. RoBERTa-Base performed best with an F1 of 0.510. The researchers noted that sarcasm and ambiguous intent often confused both human annotators and models, leading to lower performance in stance detection compared to topic classification.
Discussion: Semi-Supervised Learning and Linguistic Patterns
The inclusion of pseudo-labeled Reddit data yielded mixed results. While it significantly boosted fine-grained classification (F1 increased from 0.775 to 0.834), it degraded stance detection performance. This was attributed to a severe class imbalance in the Reddit data, where the "Favor" stance was overwhelmingly dominant, causing the model to lose its ability to distinguish minority stances.
Linguistic analysis revealed distinct "fingerprints" for different stances. "Favor" tweets frequently used proper nouns and nouns, indicating a focus on specific policies and entities. In contrast, "Against" tweets relied more on adjectives and adverbs, often employing emotional or sarcastic language to dismiss climate concerns.
Conclusion
This work demonstrates that while topic classification is relatively robust, detecting nuanced stances remains a complex challenge due to sarcasm and class imbalance. Future research should explore multi-modal and multilingual data to further refine our understanding of global climate discourse.
Summary updated