TL;DR
This paper introduces POLITISKY24, the first large-scale dataset for user-level political stance detection on the emerging decentralized social platform Bluesky, focusing on the 2024 U.S. presidential election. The dataset, comprising 16,044 user-target stance pairs with rich metadata, addresses critical gaps in political discourse analysis on newer platforms and offers a more holistic view of user opinions than traditional post-level datasets.
What Problem Does This Paper Address?
Stance detection, the task of identifying a viewpoint expressed in text towards a specific target (like a political figure), is crucial for understanding public opinion, fake news detection, and user profiling. While many datasets exist, they primarily focus on individual posts (e.g., tweets) from established platforms like Twitter. This approach misses the broader context of a user's complete posting history, which is essential for a more nuanced understanding of their political stance. Furthermore, emerging decentralized platforms like Bluesky, offering greater data accessibility, have been largely unexplored for stance detection research. The authors aim to fill this gap by creating a user-level stance dataset on Bluesky for the 2024 U.S. presidential election, specifically targeting Kamala Harris and Donald Trump.
Methods
The authors developed a sophisticated pipeline for creating the POLITISKY24 dataset. This involved:
- Data Collection: Identifying relevant U.S. political feeds on Bluesky and collecting posts over a 16-day period. Target users were defined as those with at least 10 distinct posts.
- User-Level Stance Annotation: A two-step process using Large Language Models (LLMs) was employed:
- Embedding Model Selection: A validation set was used to evaluate various embedding models for retrieving stance-relevant user posts. The KaLM-mini-v1.5 model showed superior performance.
- LLM Stance Detection: Different LLMs were tested on the validation set to identify the most effective model for user stance detection. DeepSeek-Chat-v3 achieved the highest accuracy (81.2%).
- Pipeline Configuration: The best-performing embedding model and LLM were integrated into a pipeline to label the stances of users across the entire dataset. This pipeline also generates reasoning and text spans to support the assigned stance labels, enhancing transparency.
Datasets
- POLITISKY24: The primary contribution of this paper is the introduction of the POLITISKY24 dataset. It contains 16,044 user-target stance pairs, focusing on Donald Trump and Kamala Harris for the 2024 U.S. presidential election. The dataset includes user posting histories, engagement metadata, and interaction graphs.
- Existing Datasets (for context): The paper references various existing stance detection datasets, primarily from Twitter, such as VAST, EZ-STANCE, WT-WT, and others focused on political figures and events from previous elections (2016, 2020). POLITISKY24 distinguishes itself by being user-level, on Bluesky, and for the 2024 election cycle.
Key Insights and Ideas
- User-Level Stance is Crucial: Analyzing user-level stance provides a more holistic view than post-level analysis, capturing broader patterns of expression.
- Decentralized Platforms are Underexplored: Emerging platforms like Bluesky offer unique opportunities for research due to their open nature, but remain largely untapped for tasks like stance detection.
- LLMs Can Achieve High Accuracy: The study demonstrates that LLMs, when guided by effective retrieval mechanisms, can achieve high accuracy (81%) in user-level political stance detection.
- "Neither" Stance is Challenging: The "Neither" stance proved to be the most difficult for LLMs to classify accurately, often being misclassified as "Favor." This highlights the nuance required to distinguish genuine neutrality from subtle expressions or reporting of facts without personal opinion.
- Bluesky's Left-Leaning Bias: Consistent with prior research, the analysis of the Bluesky dataset suggests a predominantly left-leaning user base, with a strong opposition to Trump and a more mixed or neutral stance towards Harris.
Key Results
- Dataset Size: POLITISKY24 contains 16,044 user-target stance pairs, derived from 8,467 users and 2,845,217 English-language posts.
- LLM Accuracy: The developed pipeline achieved an overall accuracy of 81% for stance detection.
- Model Performance: DeepSeek-Chat-v3 was identified as the best-performing LLM, achieving 81.2% overall accuracy and leading macro F1 scores for both Trump (66.2%) and Harris (72.6%). KaLM-mini-v1.5 was the top-performing embedding model.
- Error Analysis: The predominant error pattern involved misclassifying "Neither" instances as "Favor" (19.2% of errors). Key error categories included reasoning limitations, inherent subjectivity, and context retrieval issues.
Limitations and Future Work
Limitations:
- Hashtag Sampling Bias: The hashtag-based user selection method for the validation set might introduce biases, potentially overrepresenting users with extreme or clearly articulated stances.
- Left-Leaning Bias: The Bluesky platform exhibits a left-leaning bias, which is reflected in the dataset, particularly in the validation set's distribution of stances towards Trump.
- LLM Resource Constraints: The evaluation was limited to a specific set of LLMs, and findings might not generalize to all available models.
- Platform Adaptability: While the pipeline is robust, adapting it to other platforms might require adjustments, especially for seed user selection.
Future Work:
- Leveraging Graph Networks: Incorporating user graph networks to further assist with data labeling.
- Temporal Analysis: Applying the pipeline across different time windows to examine how user stances evolve over time.
- Extending Annotations: The dataset can be extended with new annotations to study the interplay between political stance and online behaviors like misinformation spread, hyper-partisan language, or hate speech.
- Network Analysis: Mapping network topology to understand political echo chambers, identify influential actors, and model narrative diffusion.
Summary updated