Interpreting Twitter User Geolocation

Ting Zhong, Tianliang Wang, Fan Zhou, Goce Trajcevski, Kunpeng Zhang, Yi Yang

Abstract

Identifying the geographic origin of social media posts is critical for applications ranging from disaster response to targeted marketing. While Graph Neural Networks (GNNs) have become the state-of-the-art for Twitter user geolocation by integrating text and social network data, they remain largely "black boxes." This paper applies influence functions to demystify these models, quantifying how individual training samples affect specific predictions. Our analysis reveals that geolocation is heavily dominated by immediate (1-hop) neighbors and that GNNs are surprisingly sensitive to individual training samples, highlighting potential adversarial vulnerabilities.

Introduction

User Geolocation (UG) on Twitter involves predicting a user's location based on their tweets and social interactions. Modern methods leverage GNNs to capture the complex relationships in "mention" networks. However, the lack of transparency in these models poses a challenge for safety-critical applications where understanding the rationale behind a prediction is as important as the prediction itself. This research seeks to provide that transparency by identifying which training users most influence a model's output for a given test user.

Methodology: Influence Functions

To avoid the prohibitive cost of retraining models to measure the impact of each training point, the authors utilize Influence Functions. This technique approximates the effect of a training sample $z$ on a test prediction by calculating the change in the model's loss.

The framework employs:

  • Hessian-Vector Products (HVPs): Efficiently approximating the inverse Hessian matrix using conjugate gradients to handle large-scale social graphs.
  • Model Agnosticism: Testing on both Simple Graph Convolution (SGC) and Multi-Layer Perceptron (MLP) architectures to compare graph-based and non-graph-based behaviors.

Experimental Setup

The study uses two benchmark datasets:

  1. GeoText: A smaller dataset with approximately 5,700 training users and 129 regional labels.
  2. Twitter-US: A large-scale dataset with approximately 429,000 training users and 256 regional labels. Both datasets provide tweet content and a mention network for graph construction.

Analysis of Model Influence

The application of influence functions yielded several key insights into the internal mechanics of geolocation models.

Comprehensive analysis of influence in Twitter geolocation models, illustrating n-hop neighbor impact, accuracy breakdown, and distance correlation.

The Dominance of Local Neighborhoods

A primary discovery is the "1-hop" dominance: influence is almost entirely concentrated in a user's immediate neighbors. As shown in the scatter plots of the composite analysis above, influence scores drop sharply for 2-hop and 3-hop neighbors. This suggests that while the network view is essential, higher-order connectivity may provide diminishing returns for geolocation tasks.

Geographic Proximity and the 10km Threshold

Influence is strongly correlated with physical distance. The impact of a training sample on a test prediction remains high within a 10km radius but declines significantly beyond this threshold. This confirms that "in-region" samples—those geographically and topologically close—are the primary drivers of accuracy.

GNN Sensitivity and Negative Influence

The SGC model demonstrated higher sensitivity to individual training samples compared to the MLP. While this sensitivity contributes to the GNN's superior performance, it also indicates a higher susceptibility to adversarial perturbations. Furthermore, the analysis identified "negative influence" nodes—training samples that actually degrade test accuracy. The MLP model contained significantly more of these confusing nodes, explaining its lower overall performance relative to the GNN.

Key Results

Despite the complexity of GNNs, the authors found a performance plateau where SGC, GCN, and MLP models achieved similar accuracy (approximately 60-62% within 161km). This suggests that current benchmarks may be reaching their limits, and further gains might require more than just increasing model depth or complexity.

Conclusion and Future Work

This work provides a post-hoc explanation framework that sheds light on the "black box" of GNN-based geolocation. By identifying the critical role of 1-hop neighbors and the presence of negative influence nodes, it paves the way for more robust and efficient models. Future research will focus on developing GNNs that are less sensitive to individual node perturbations and using influence functions to filter training data for improved accuracy.

Summary updated

BibTeX
@inproceedings{zhong2020interpreting,
  author = {Ting Zhong and Tianliang Wang and Fan Zhou and Goce Trajcevski and Kunpeng Zhang and Yi Yang},
  title = {{Interpreting Twitter User Geolocation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.79},
}
Figure