Geocoding Without Geotags: A Text-based Approach for reddit

Keith HarrigianEMNLP 2018

TL;DR

This paper introduces the first method for geolocating Reddit users based solely on their text content, overcoming the platform's inherent pseudonymity and lack of explicit location data. By developing a novel heuristic labeling procedure and leveraging subreddit metadata, they achieve competitive geolocation accuracy, demonstrating that rich user location information can be inferred even without geotags.

What Problem Does This Paper Address?

Social media platforms offer a treasure trove of data for understanding demographics and user behavior. However, many platforms, like Reddit, prioritize user privacy through pseudonymity and don't collect explicit location data (like geotags or profile fields). This makes traditional supervised learning approaches for geolocation—which rely on labeled data—extremely difficult to implement and validate. Existing geolocation methods, often developed for platforms like Twitter, may not transfer well to Reddit due to differences in user demographics and platform-specific features. The challenge, therefore, is to develop a robust method for inferring user locations on platforms like Reddit, where direct location information is absent.

Methods

The core of this paper lies in its innovative approach to creating a labeled dataset for Reddit.

  • Heuristic Labeling Procedure: The authors devised a text-based heuristic schema to generate ground truth location labels. This involved:
  • Seed Submission Querying: Searching for Reddit posts with titles like "Where are you from?" or "Where do you live?".
  • Comment Filtering: Carefully filtering comments to ensure they represented self-identified home locations, excluding replies and mentions of moving or being born/raised elsewhere.
  • Entity Extraction: Using a gazetteer (GEONAMES data) and an abbreviation dictionary to identify location names within comments. They employed a multi-token n-gram chunking approach and a rule to capture "City, State, Country" patterns.
  • Geocoding: Utilizing Google's Geocoding API to convert extracted location strings into coordinate pairs, with manual mapping of region biases to subreddits to aid disambiguation.
  • Geographic Representation: Aggregating extracted locations per user, using the geodesic median for locations above city-level resolution to determine the true coordinate pair.
  • Geolocation Inference Model: They adapted a generative model (based on Cheng et al., 2010) that incorporates:
  • Word Usage (w): Concatenated text from all user comments.
  • Subreddit Submissions (s): Frequency distribution of subreddits a user posts in.
  • Posting Frequency (τ): A 24-dimensional vector representing hourly posting habits (UTC).
  • They used a Dirichlet Process Mixture Model (DPMM) for probability density estimation to handle feature sparsity.

Datasets

  • Reddit Data: A new dataset was created by the authors, consisting of 96,071 comments from 89,697 unique users, labeled using their heuristic procedure. This dataset is split into US-restricted and GLOBAL versions.
  • Benchmark Twitter Datasets:
    • GEOTEXT (Eisenstein et al., 2010)
    • TWITTER-US (Roller et al., 2012)
    • TWITTER-WORLD (Han et al., 2012) These datasets were compiled from Twitter's streaming API and contain geotagged comments.

Key Insights and Ideas

  • Pseudonymity is Not an Insurmountable Barrier: The paper strongly demonstrates that even on highly pseudonymous platforms like Reddit, valuable demographic information like location can be inferred from text alone.
  • Subreddit Metadata is Highly Predictive: Platform-specific features, such as the subreddits a user participates in, are crucial for improving geolocation accuracy, especially when compared to models that only use general text content. This highlights the importance of understanding user communities.
  • Domain-Specific Training is Superior: Models trained on data from a specific platform (e.g., Reddit) significantly outperform models trained on one platform and applied to another (e.g., Twitter-trained models on Reddit). This is attributed to the unique characteristics and metadata of each platform.
  • Temporal Patterns Offer Value, Especially Globally: While temporal posting habits (hourly activity) don't significantly boost accuracy for US-only datasets, they become important for global datasets, likely due to timezone variations.
  • Heuristic Labeling is Viable: The paper validates a novel, albeit imperfect, method for creating ground truth location labels for platforms lacking explicit geotags, opening the door for supervised learning in similar contexts.

Key Results

  • Labeling Accuracy: The heuristic labeling procedure achieved a precision of 96.6% for correctly extracting location names, with 92.55% of these labels being at the correct resolution.
  • Within-Domain Performance: The authors' model trained on Reddit data achieved a median error of 157 miles for US users and 266 miles for international users.
  • Cross-Domain Performance: Models trained on Reddit data outperformed baselines on Twitter datasets in most cases, demonstrating the robustness of their approach. However, transferring models from Twitter to Reddit resulted in a significant loss of performance, particularly due to the lack of subreddit metadata in the Twitter training data.
  • Feature Impact: Subreddit metadata consistently improved performance on both US and GLOBAL Reddit datasets. Temporal features improved performance on the GLOBAL dataset and helped mitigate performance loss during cross-domain transfer.

Limitations and Future Work

  • Labeling Imperfections: The authors acknowledge that their labeling procedure is not perfect. Issues include disambiguation errors by the geocoding API, users referencing non-home locations, and the exclusion of cities not in their gazetteer or captured by their rules. Future work could explore more robust entity resolution systems.
  • Bias in Labeled Data: The authors plan to investigate underlying biases in their labeled user population, examining differences in user activity, interests, and topicality compared to the general Reddit population.
  • Representation of Non-North American Users: The current dataset tends to over-index on North American users. Future work will explore different seed-submission sampling methods to improve the representation of users from other regions.
  • Model Complexity: The paper suggests that more complex, multi-modal architectures (like those used in state-of-the-art Twitter geolocation) could further improve performance on Reddit, especially by better leveraging subreddit metadata alongside text.

Summary updated

BibTeX
@inproceedings{harrigian2018geocoding,
  author = {Keith Harrigian},
  title = {{Geocoding Without Geotags: A Text-based Approach for reddit}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure