Measuring Social Solidarity During Crisis: The Role of Design Choices

Steffen Eger, Dan Liu, Daniela GrunowJ. Soc. Comput 2022 · OpenReview

Abstract

Building on previous research, this study assesses how social solidarity towards migrants and refugees shifted before and after the onset of the COVID-19 pandemic. We analyzed a novel longitudinal dataset of over 240,000 migration-related tweets collected between September 2019 and June 2021. By training a BERT model using expert and crowd-sourced annotations alongside data augmentation strategies, we achieved performance near the human upper bound. Our findings reveal that while migrant solidarity became increasingly salient and contested during the early stages of the pandemic, it declined by late 2020. Crucially, we demonstrate that the perceived ratio of solidarity to anti-solidarity is heavily influenced by "design choices," such as tweet language, user anonymity, and sampling methods.

Introduction

Social scientists and policymakers frequently utilize social media to gauge public sentiment during crises. However, existing research often suffers from several critical "blind spots":

  1. Lack of Baselines: Studies frequently begin only after a crisis starts, making it impossible to distinguish new behaviors from pre-existing trends.
  2. One-Sided Focus: Literature often prioritizes anti-social behavior (e.g., hate speech) over the interplay between pro-social (solidarity) and anti-social expressions.
  3. Short Timeframes: Brief observation windows miss the long-term evolution of social dynamics.
  4. Sampling Bias: Choices regarding language or hashtags can inadvertently bake bias into conclusions.

This paper addresses these gaps by providing a rare "pre-crisis" baseline and analyzing the long-term development of migrant-related discourse through the COVID-19 era.

Methodology

We employed a multi-stage pipeline to bridge the gap between small-scale human annotation and large-scale automated analysis. Initially, ~2,000 tweets were categorized by experts and crowd workers into solidarity, anti-solidarity, ambivalent, or not applicable.

To scale this, we utilized multilingual BERT (mBERT) and XLM-R for 3-way classification. To reach human-level performance, we implemented sophisticated data augmentation strategies, including back-translation (creating synthetic training variations) and self-learning. In the latter, an ensemble of models labeled 105,000 additional tweets; those with high agreement served as a "silver standard" to retrain the final model, boosting the Macro-F1 score from ~60% to ~80%.

Dataset Characteristics

The final extended dataset comprises 243,812 tweets. As shown in the table below, the majority of the discourse was in German (61.2%) and focused specifically on refugees (94.5%).

Table 1: Composition of the longitudinal dataset across language, topic, country, and predicted sentiment classes.

Our analysis shows that migrant solidarity spiked during the early stages of the pandemic (March 2020) but declined steadily thereafter. By summer 2021, tweet volumes had fallen slightly below pre-pandemic levels. Interestingly, peaks in solidarity were often tied to specific humanitarian events—such as the fire at the Moria refugee camp in September 2020—rather than the pandemic itself.

Figure 1: Distribution of migration-related tweets by country and date (Sept 2019 – June 2021), highlighting volume spikes during major crises.

While the COVID-19 pandemic "infected" the migration discourse in March 2020 (with over 40% of migration tweets mentioning the virus), this association was short-lived, returning to low levels by summer 2020 even as the health crisis persisted.

The Role of Design Choices

A central contribution of this work is identifying how sampling design alters findings:

  • The "Solidarity Bias" of Hashtags: Tweets without hashtags are significantly more likely to express anti-solidarity. Sampling only by popular hashtags (e.g., #RefugeesWelcome) leads to an over-reporting of social solidarity.
  • Anonymity and Hostility: "Anonymous" users (those without location data) writing in German were more likely to post anti-solidarity content than identified users.
  • Language as Identity: In Germany, English-language tweets exhibited much higher levels of solidarity than native German tweets, suggesting that language choice is intertwined with national identity and education levels.

Conclusion

These findings highlight the necessity of long-term observation and careful sampling in crisis research. While social media offers a window into public sentiment, researchers must account for the biases inherent in their design choices. Future work should integrate representative survey data to further calibrate social media findings and explore dynamics across different platforms.

Summary updated

Abstract

Building on our previous work, we assess how social solidarity towards migrants and refugees has changed before and after the onset of the COVID-19 pandemic, by collecting and analyzing a large, novel, and longitudinal dataset of migration-related tweets. To this end, we first annotate above 2000 tweets for (anti-) solidarity expressions towards immigrants, utilizing two annotation approaches (experts vs. crowds). On these annotations, we train a BERT model with multiple data augmentation strategies, which performs close to the human upper bound. We use this high-quality model to automatically label over 240 000 tweets between September 2019 and June 2021. We then assess the automatically labeled data for how statements related to migrant (anti-)solidarity developed over time, before and during the COVID-19 crisis. Our findings show that migrant solidarity became increasingly salient and contested during the early stages of the pandemic but declined in importance since late 2020, with tweet numbers falling slightly below pre- pandemic levels in summer 2021. During the same period, the share of anti-solidarity tweets increased in a sub- sample of COVID-19-related tweets. These findings highlight the importance of long-term observation, pre- and post-crisis comparison, and sampling in research interested in crisis related effects. As one of our main contributions, we outline potential pitfalls of an analysis of social solidarity trends: for example, the ratio of solidarity and anti-solidarity statements depends on the sampling design, i.e., tweet language, Twitter-user accounts' national identification (country known or unknown) and selection of relevant tweets. In our sample, the share of anti-solidarity tweets is higher in native (German) language tweets and among “anonymous” Twitter users writing in German compared to English-language tweets of users located in Germany.
BibTeX
@article{eger2022measuring,
  author = {Steffen Eger and Dan Liu and Daniela Grunow},
  title = {{Measuring Social Solidarity During Crisis: The Role of Design Choices}},
  journal = {J. Soc. Comput.},
  year = {2022},
  url = {https://openreview.net/forum?id=CkidX0KSrJT},
}
Figure