Deriving Disinformation Insights from Geolocalized Twitter Callouts

David Tuxworth, Dimosthenis Antypas, Luis Espinosa-Anke, Jose Camacho-Collados, Alun Preece, David RogersarXiv 2021 · arXiv

TL;DR

This paper introduces a two-stage framework for analyzing disinformation on Twitter across multiple languages and geographic regions. By combining BERT-based geolocation with Word2vec embedding analysis, the researchers successfully mapped how disinformation narratives—ranging from Brexit to COVID-19 conspiracies—evolve differently in European versus non-European contexts.

What Problem Does This Paper Address?

Disinformation is a global, multilingual phenomenon, but its narratives are often deeply rooted in local culture and politics. For instance, a "fake news" claim about immigration might look very different in the UK compared to the US.

The challenge for researchers is twofold: first, only a tiny fraction (about 0.34%) of tweets contain precise GPS metadata, making it hard to segment data by region. Second, the sheer volume of social media data makes it difficult to track subtle shifts in language and context over time. This paper aims to bridge these gaps by providing a scalable way to geolocate tweets using text-based metadata and then extract semantic insights using natural language processing (NLP).

Methods

The authors employ a two-stage pipeline:

  1. Geospatial Classification: Since most tweets lack GPS data, the authors trained BERT-based classifiers to categorize tweets as "European" or "non-European." They used the small subset of tweets that did have GPS data as "ground truth" seeds, training the model on user-provided location and description fields.
  2. Embedding-Based Analysis: Once the data was segmented by region and year (2019 vs. 2020), the authors used Word2vec (CBOW) to create language models for each subset. They also utilized lexical specificity, a statistical measure that identifies the most representative words for a specific corpus compared to a reference set.

This approach allows for "analogical reasoning" (e.g., London is to Britain as [?] is to America) to see how different regions perceive media outlets or political figures.

Datasets

The study is built upon a massive collection of Twitter data:

  • Initial Collection: Approximately 88 million tweets collected between 2019 and 2020 using 97 keywords indicative of disinformation (e.g., "fake news," "propaganda").
  • Training Set: 294,877 tweets containing geolocation metadata were used to train the BERT classifiers.
  • Final Analysis Set: A filtered subset of 36,655,061 tweets in English, French, and Spanish.
Figure 1: Number of tweets by year and language.

Key Insights and Ideas

  • Regional Narrative Divergence: The models captured distinct regional "flavors" of disinformation. For the query "immigrant" in 2020, the European English model associated the term with "Greeks" and "Europeans," while the non-European model associated it with "Mexicans" and "Blacks."
  • Temporal Shifts: In 2019, the English European discourse was dominated by Brexit. By 2020, this had shifted almost entirely to generic politics and COVID-19.
  • Conspiracy Detection: The Word2vec models successfully identified emerging conspiracy theories. In 2020, the term "vaccine" became semantically linked to "microchip," "rfid," and "bill_gates," reflecting the rise of pandemic-related disinformation.
  • Monolingual vs. Multilingual: Interestingly, the authors found that monolingual BERT models generally outperformed a single multilingual BERT (mBERT) model for these high-resource languages, suggesting that language-specific nuances are still better captured by dedicated models.

Key Results

The BERT-based geolocation method proved highly effective, even with limited training data:

  • Accuracy: All yearly BERT models achieved over 85% accuracy.
  • Performance: The English model performed best with a 92% F1-score, while the French model was the "worst" but still respectable at 86-87%.
  • Robustness: Classifiers trained on 2019 data remained robust when tested on 2020 data, showing that user metadata (location/description) is a stable feature for geolocation.
Table 3: Classification results for the 2019 and 2020 datasets for each language model. Evaluation metrics: accuracy and macro- averaged precision, recall and F1. mBERT* model is trained on the whole corpus including the three languages. Naive baseline refers to a system where every tweet entry is classified as European

Limitations and Future Work

The authors acknowledge several areas for improvement:

  • Granularity: The current model only distinguishes between European and non-European regions. Future work aims to classify data at the country level.
  • User Noise: User-defined location fields are often "noisy" or inaccurate (e.g., someone listing their location as "The Moon"), though the BERT models proved surprisingly resilient to this.
  • Accessibility: The researchers are working on turning these analytical methods into "dashboard" tools, allowing subject-matter experts without deep technical training to query the data and visualize semantic shifts using techniques like t-SNE.

Summary updated

Abstract

This paper demonstrates a two-stage method for deriving insights from social media data relating to disinformation by applying a combination of geospatial classification and embedding-based language modelling across multiple languages. In particular, the analysis in centered on Twitter and disinformation for three European languages: English, French and Spanish. Firstly, Twitter data is classified into European and non-European sets using BERT. Secondly, Word2vec is applied to the classified texts resulting in Eurocentric, non-Eurocentric and global representations of the data for the three target languages. This comparative analysis demonstrates not only the efficacy of the classification method but also highlights geographic, temporal and linguistic differences in the disinformation-related media. Thus, the contributions of the work are threefold: (i) a novel language-independent transformer-based geolocation method; (ii) an analytical approach that exploits lexical specificity and word embeddings to interrogate user-generated content; and (iii) a dataset of 36 million disinformation related tweets in English, French and Spanish.
BibTeX
@misc{tuxworth2021deriving,
  author = {David Tuxworth and Dimosthenis Antypas and Luis Espinosa-Anke and Jose Camacho-Collados and Alun Preece and David Rogers},
  title = {{Deriving Disinformation Insights from Geolocalized Twitter Callouts}},
  year = {2021},
  eprint = {2108.03067},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2108.03067},
}
Figure