Measuring the Salad Bowl: Superdiversity on Twitter

Laura Pollacci, Alina Sîrbu, Fosca Giannotti, Dino PedreschiarXiv 2022 · DOI · arXiv

Introduction

"Superdiversity" describes the intricate layers of cultural diversity in modern populations, driven by complex immigration patterns. While understanding this diversity is crucial for effective policy-making and social integration, traditional measurement methods are often inadequate. National censuses are infrequent, expensive, and typically only capture static metrics like "country of birth," failing to reflect the dynamic cultural interplay within a community.

Researchers have proposed a new approach: using social media data to track cultural diversity in real-time. By analyzing how the emotional meaning of words shifts within local Twitter communities, they have developed the Superdiversity Index (SI) to bridge the gap between official statistics and the fast-moving reality of migration.

The Superdiversity Index (SI) Framework

The core innovation of this research is the hypothesis that people from different cultural backgrounds associate distinct emotional "valences" (positive or negative weights) with the same words. To quantify this, the authors developed the Emotional Spreading Algorithm (EmoSA).

The EmoSA workflow for calculating the Superdiversity Index (SI) by comparing community-derived lexicons with standard valences.

As shown in the workflow above, the process involves:

  1. Network Building: Creating a co-occurrence network of words from geolocalized tweets.
  2. Epidemic Spreading: Using a small "seed" of human-tagged emotional words (from the ANEW lexicon) to spread emotional values to neighboring words in the network.
  3. Filtering: Ensuring accuracy by only adopting values where neighbors show a consistent emotional distribution.
  4. Calculation: The SI is defined as the distance (1 minus the Pearson correlation) between this community-derived lexicon and a standard "ground truth" lexicon. A higher SI indicates a community whose language usage deviates significantly from the standard, suggesting higher cultural diversity.

Data and Methodology

The study utilized a massive multi-modal data stack, including approximately 73 million geolocalized tweets from the UK and Italy collected between August and October 2015. This data was cross-referenced with validation lexicons (ANEW, SentiWordNet) and ground truth immigration data from the European Commission’s Joint Research Center (D4I dataset).

Summary of the processed Twitter datasets and the number of matched cities for the UK and Italy.

Key Results and Insights

The Superdiversity Index demonstrated a remarkably high correlation with official immigration statistics, outperforming simpler metrics like "lexical richness" or "language entropy" (the number of different languages spoken).

  • High Accuracy: In the UK, the SI achieved a Pearson correlation of 0.94 at the regional level. In Italy, correlations frequently exceeded 0.85 across various geographical resolutions.
  • The "Salad Bowl" Effect: Unlike the "melting pot" theory where cultures fuse, the "salad bowl" concept suggests cultures coexist while maintaining distinct qualities. The SI captures the subtle ways these distinct components influence the shared language of a region.
  • Robustness: A "null model" test—reshuffling tweets randomly across regions—destroyed the correlation, proving that the SI captures unique local community characteristics rather than random noise.

Limitations and Future Work

While the results are promising, certain regions like London and Northeast England acted as outliers due to extreme linguistic diversity or strong local dialects. Future iterations of the index will likely require "correction factors" for population density and local dialects to ensure comparability across diverse regions.

The ultimate goal is "nowcasting": using the SI to provide real-time estimates of migration stocks in areas where official census data is missing or outdated, offering a dynamic tool for sociologists and policymakers alike.

Summary updated

Abstract

Superdiversity refers to large cultural diversity in a population due to immigration. In this paper, we introduce a superdiversity index based on the changes in the emotional content of words used by a multi-cultural community, compared to the standard language. To compute our index we use Twitter data and we develop an algorithm to extend a dictionary for lexicon-based sentiment analysis. We validate our index by comparing it with official immigration statistics available from the European Commission's Joint Research Center, through the D4I data challenge. We show that, in general, our measure correlates with immigration rates, at various geographical resolutions. Our method produces very good results across languages, being tested here both on English and Italian tweets. We argue that our index has predictive power in regions where exact data on immigration is not available, paving the way for a nowcasting model of immigration rates.
BibTeX
@misc{pollacci2022measuring,
  author = {Laura Pollacci and Alina Sîrbu and Fosca Giannotti and Dino Pedreschi},
  title = {{Measuring the Salad Bowl: Superdiversity on Twitter}},
  year = {2022},
  eprint = {2204.10646},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.10646},
}
Figure