Abstract
Social solidarity is a fundamental pillar of the European Union, yet traditional survey-based methods for measuring it are often slow and prone to social desirability bias. This study presents a high-performing NLP pipeline designed to track European solidarity on Twitter in real-time. By leveraging a hybrid expert-crowd annotation strategy and advanced data augmentation—including back-translation and self-learning—the researchers achieved an 84.5% macro-F1 score in a three-way classification task (Solidarity, Anti-Solidarity, and Other). Analysis of over 270,000 tweets reveals that while pro-solidarity discourse remained dominant during the COVID-19 pandemic, "anti-solidarity" sentiment experienced sharp spikes tied to lockdowns and humanitarian crises, settling at a higher baseline by late 2020.
Introduction
Social solidarity—the willingness to share resources and support others—is often described as the "glue" that holds the European Union together. Historically, social scientists have relied on surveys to gauge public sentiment regarding solidarity. However, these methods are expensive, infrequent, and often fail to capture extreme or unpopular opinions due to social desirability bias.
Social media platforms like Twitter offer a real-time alternative for observing public discourse. However, existing Natural Language Processing (NLP) tools frequently conflate solidarity with general "positive sentiment." This research bridges the gap between social science theory and machine learning by developing a robust framework to identify both solidarity and its active contestation, termed "anti-solidarity."
Defining Solidarity and Anti-Solidarity
The researchers defined the problem as a three-way classification task: Solidarity, Anti-Solidarity, and Other (which includes ambivalent or non-applicable content). To ensure the model captured the nuances of political discourse, the team utilized a rigorous annotation process.
One of the study's critical findings was that solidarity is not synonymous with positive sentiment. The correlation between traditional sentiment analysis (like VADER) and solidarity labels was remarkably low (~0.2). For instance, a tweet criticizing the EU for failing to provide enough aid to a member state might have a negative sentiment score but would be labeled as "pro-solidarity" because it advocates for increased support.
Methodology and Data Augmentation
The study utilized a multi-stage pipeline to overcome the challenges of small human-labeled datasets:
- Hybrid Annotation: A 6-stage expert process refined the guidelines, followed by a "crowd experiment" involving 125 NLP students to scale the dataset to 2,300 labeled English and German tweets.
- Model Architecture: The team benchmarked Multilingual BERT (MBERT) and XLM-R, ultimately using an ensemble of the top 15 models.
- Data Augmentation "Secret Sauce": To expand the training set, the authors employed:
- Back-translation: Creating synthetic variations by translating tweets to a pivot language and back.
- Domain Fine-tuning: Training models on masked language modeling using a specific 270k tweet corpus.
- Self-Learning: An ensemble of 9 models auto-labeled the unlabeled corpus, retaining only high-confidence samples where at least 7 models agreed. This expanded the training set by 105,000 samples.
Temporal Trends and Crisis Escalation
The large-scale analysis of 271,930 tweets from September 2019 to December 2020 revealed that solidarity and anti-solidarity are highly dynamic, responding sharply to "incisive political events."
As shown in the temporal analysis, anti-solidarity sentiment spiked during the first European lockdowns in March 2020 and again during the Moria refugee camp fire in September 2020. Notably, there was a significant correlation (0.45–0.47) between COVID-19 infection rates and the volume of anti-solidarity tweets. While pro-solidarity discourse generally outnumbered anti-solidarity 2-to-1, the latter did not return to pre-pandemic levels, suggesting a lasting increase in polarization.
Key Results and Performance
The baseline BERT model, trained only on expert data, achieved a 58% macro-F1. However, the fully augmented ensemble reached 84.5% macro-F1, nearly matching the agreement levels of human experts. This performance leap demonstrates the efficacy of combining expert knowledge with large-scale automated data augmentation.
Conclusion and Future Work
This study demonstrates that NLP can effectively track complex sociopolitical constructs like solidarity. While the research was limited to English and German and focused on specific hashtags related to migration and finance, it provides a scalable framework for future studies. Expanding the dataset to include more European languages and investigating the causal links between policy decisions and online sentiment remain vital areas for future exploration. Despite the rise in contestation, the continued dominance of pro-solidarity discourse suggests that the "European project" maintained significant public support throughout the pandemic's most challenging periods.
Summary updated