Observational Comparison of Geo-tagged and Randomly-drawn Tweets

Tom Lippincott, Annabelle CarrellNAACL 2018

TL;DR

This paper investigates how geo-tagged tweets differ from randomly sampled tweets across various languages and geographies, revealing that geo-tagged users are more active and engaged. The study also finds that using the "description" field for extracting user self-identification is more precise and yields more results than complex regular expressions.

What Problem Does This Paper Address?

Twitter's geo-tagging feature offers a valuable source of real-time, location-specific information for various applications, from bot detection to disaster awareness. However, only a fraction of users opt-in to geo-tagging, and these users might systematically differ from the general Twitter population. Previous research has primarily focused on English-language tweets within the US and UK, leaving a gap in understanding these differences across a broader range of languages and geographies. This paper aims to bridge this gap by conducting an observational comparison of geo-tagged (GT) and randomly drawn (NGT) tweets, exploring various linguistic and demographic characteristics.

Methods

The researchers collected a geo-tagged (GT) dataset from the final week of November 2017 and a non-geo-tagged (NGT) dataset representing a 1% uniform random sample from the same period. They indexed these tweets and user JSON objects in Elasticsearch for comparison. Key categorical fields examined include language, hashtag, user mention, and time zone.

For user self-identification, they adopted a method from prior work, extracting roles based on the regular expression "I('m|am) an? (\S+)" and also by querying users whose "description" field contained the role. They found that querying the "description" field, despite its simplicity, offered higher precision and yielded significantly more results than complex regular expressions, making it the preferred method for demographic information extraction.

To compare distributions of discrete outcomes (like language use, role occurrence), they calculated the Jensen-Shannon Divergence (JSD), a symmetric measure of difference between probability distributions. To explore variations across languages, they calculated the Spearman rank correlation coefficient over the JSD values.

Datasets

  • Geo-tagged (GT) dataset: Collected from the final week of November 2017.
  • Non-geo-tagged (NGT) dataset: A 1% uniform random sample from the same period, excluding geo-tagged content.

The paper notes that Twitter's terms of service prevent the distribution of the underlying data, but they make the fine-grained counts available as pickled query results from ElasticSearch.

Key Insights and Ideas

  • Description field reigns supreme for user roles: Querying the "description" field for user self-identification (roles) proved to be significantly more precise and yielded orders of magnitude more results than using complex regular expressions on tweet text. This is a crucial finding for researchers aiming to extract demographic information efficiently.
  • Geo-tagged users are more active: GT users tweet at a significantly higher rate (over triple that of NGT users) and appear to be more engaged with Twitter's community structure, exhibiting higher average friend and follower counts.
  • Language skews in geo-tagging: While English and Portuguese show high proportions of geo-tagging, languages like Japanese, Arabic, Thai, and particularly Korean have low proportions.
  • Institutional accounts dominate GT hashtags and mentions: Hashtags and user mentions in GT tweets are heavily skewed towards institutional accounts (companies, news organizations, sports franchises) and discussions related to potential employment, suggesting a strong presence of institutional or spam accounts in geo-tagged data.
  • Pop stars vs. political figures: NGT users are more likely to mention pop stars, while GT users tend to mention institutional accounts and, notably, political figures like Donald Trump.
  • Performance roles are more public: Roles associated with performance (singer, actor, athlete, cheerleader) seem more inclined to be geo-tagged, while religious roles (Christian, atheist) and musical fandoms (Belieber, Directioner) show a preference against geo-tagging.

Key Results

  • Tweet Volume: GT users tweet at over triple the rate (8.4 tweets/user) compared to NGT users (2.5 tweets/user).
  • Account Age: GT accounts tend to be about twice the age of NGT accounts.
  • Verified Accounts: 1% of GT users are verified, compared to 0.5% of NGT users.
  • User Behavior: GT users show higher average counts for friends, followers, and favorites compared to NGT users.
  • Role Extraction Precision: The "description" field method for role extraction achieved higher precision than the "IamA" pattern.
  • Spearman Correlation for Roles: A high Spearman correlation of 0.944 was found between the relative frequencies of roles in GT and NGT datasets, indicating that the types of roles are similar, even if their prevalence differs.
  • Spearman Correlation for Distributions: Pairwise Spearman correlations between JSD values for hashtag, user mention, and time zone distributions across languages showed positive associations, with user mention and hashtag variations being more correlated with each other than with time zone.

Limitations and Future Work

  • Data Access: Twitter's terms of service limit the distribution of raw data, necessitating the sharing of aggregated counts.
  • Time Zone Interpretation: The paper acknowledges that a better understanding of how time zones are set (e.g., manual vs. automatic) would aid in interpreting time zone data.
  • Recall Measurement: The study focused on precision for role extraction due to the difficulty of measuring recall, especially for low-frequency roles. Measuring recall is left for future work.
  • Supervised Learning: The authors plan to exploit the findings on user descriptions as supervised input for discriminative models to extract unconstrained self-identification.
  • Beyond English: Future work will extend experiments to languages beyond English and explore correlations between regional/language-specific variations and cultural/political axes.
  • Content Analysis: The study did not directly examine tweet text content beyond role extraction. Future work could extend to variation over simple lexical features.

Summary updated

BibTeX
@inproceedings{lippincott2018observational,
  author = {Tom Lippincott and Annabelle Carrell},
  title = {{Observational Comparison of Geo-tagged and Randomly-drawn Tweets}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
}
Figure