TL;DR
This paper introduces a two-stage framework for analyzing disinformation on Twitter across multiple languages and geographic regions. By combining BERT-based geolocation with Word2vec embedding analysis, the researchers successfully mapped how disinformation narratives—ranging from Brexit to COVID-19 conspiracies—evolve differently in European versus non-European contexts.
What Problem Does This Paper Address?
Disinformation is a global, multilingual phenomenon, but its narratives are often deeply rooted in local culture and politics. For instance, a "fake news" claim about immigration might look very different in the UK compared to the US.
The challenge for researchers is twofold: first, only a tiny fraction (about 0.34%) of tweets contain precise GPS metadata, making it hard to segment data by region. Second, the sheer volume of social media data makes it difficult to track subtle shifts in language and context over time. This paper aims to bridge these gaps by providing a scalable way to geolocate tweets using text-based metadata and then extract semantic insights using natural language processing (NLP).
Methods
The authors employ a two-stage pipeline:
- Geospatial Classification: Since most tweets lack GPS data, the authors trained BERT-based classifiers to categorize tweets as "European" or "non-European." They used the small subset of tweets that did have GPS data as "ground truth" seeds, training the model on user-provided location and description fields.
- Embedding-Based Analysis: Once the data was segmented by region and year (2019 vs. 2020), the authors used Word2vec (CBOW) to create language models for each subset. They also utilized lexical specificity, a statistical measure that identifies the most representative words for a specific corpus compared to a reference set.
This approach allows for "analogical reasoning" (e.g., London is to Britain as [?] is to America) to see how different regions perceive media outlets or political figures.
Datasets
The study is built upon a massive collection of Twitter data:
- Initial Collection: Approximately 88 million tweets collected between 2019 and 2020 using 97 keywords indicative of disinformation (e.g., "fake news," "propaganda").
- Training Set: 294,877 tweets containing geolocation metadata were used to train the BERT classifiers.
- Final Analysis Set: A filtered subset of 36,655,061 tweets in English, French, and Spanish.
Key Insights and Ideas
- Regional Narrative Divergence: The models captured distinct regional "flavors" of disinformation. For the query "immigrant" in 2020, the European English model associated the term with "Greeks" and "Europeans," while the non-European model associated it with "Mexicans" and "Blacks."
- Temporal Shifts: In 2019, the English European discourse was dominated by Brexit. By 2020, this had shifted almost entirely to generic politics and COVID-19.
- Conspiracy Detection: The Word2vec models successfully identified emerging conspiracy theories. In 2020, the term "vaccine" became semantically linked to "microchip," "rfid," and "bill_gates," reflecting the rise of pandemic-related disinformation.
- Monolingual vs. Multilingual: Interestingly, the authors found that monolingual BERT models generally outperformed a single multilingual BERT (mBERT) model for these high-resource languages, suggesting that language-specific nuances are still better captured by dedicated models.
Key Results
The BERT-based geolocation method proved highly effective, even with limited training data:
- Accuracy: All yearly BERT models achieved over 85% accuracy.
- Performance: The English model performed best with a 92% F1-score, while the French model was the "worst" but still respectable at 86-87%.
- Robustness: Classifiers trained on 2019 data remained robust when tested on 2020 data, showing that user metadata (location/description) is a stable feature for geolocation.
Limitations and Future Work
The authors acknowledge several areas for improvement:
- Granularity: The current model only distinguishes between European and non-European regions. Future work aims to classify data at the country level.
- User Noise: User-defined location fields are often "noisy" or inaccurate (e.g., someone listing their location as "The Moon"), though the BERT models proved surprisingly resilient to this.
- Accessibility: The researchers are working on turning these analytical methods into "dashboard" tools, allowing subject-matter experts without deep technical training to query the data and visualize semantic shifts using techniques like t-SNE.
Summary updated