Introduction
Twitter-Demographer is an open-source, flow-based tool designed to help researchers "hydrate" Twitter IDs and enrich them with demographic, sentiment, and location metadata. For social scientists and NLP researchers, raw tweet text is rarely enough to understand societal trends or control for confounding variables. Researchers typically require metadata such as user location, age, and message sentiment or toxicity.
Obtaining this information is often a logistical challenge, requiring researchers to navigate the Twitter API, manage rate limits, and chain together various machine learning models with conflicting dependencies. Furthermore, researchers must ensure that the resulting dataset complies with user privacy and GDPR regulations. Twitter-Demographer addresses these issues by providing a unified, extensible, and reproducible framework for data enrichment using a modular pipeline approach.
Methodology
The tool is built on the flow-based programming paradigm, where the data enrichment process is treated as a factory line of "black-box" components. Each component, such as a sentiment classifier or a location decoder, has a defined set of required inputs (e.g., "text") and generated outputs (e.g., "sentiment_score").
A central Demographer object manages these components, verifying compatibility and handling the execution flow. To ensure robustness, the tool includes automatic caching and checkpointing, allowing processes to restart from the last successful step in case of network failures or API rate limits. Additionally, the tool utilizes Python decorators (such as @not_null) to handle data quality issues, ensuring the pipeline remains stable during long-running jobs.
Data Enrichment and Privacy
Twitter-Demographer integrates several pre-trained models and benchmarks, including the M3 classifier for age and gender inference, the Perspective API for toxicity scoring, and HuggingFace Transformers for sentiment and topic modeling.
To comply with Twitter's Terms of Service, which often restrict sharing full tweet objects, the tool automates the "hydration" process. It converts lists of tweet IDs back into full objects using the Twitter API v2, managing batching and rate-limiting logic internally.
Privacy-by-Design
To protect user anonymity and ensure GDPR compliance, the tool implements several "privacy-by-design" features:
- Unidirectional Hashing: Screen names are replaced with consistent but non-reversible hash codes.
- Label Swapping: A subset of inferred labels (e.g., age) is randomly swapped between instances. This preserves aggregate statistics for the dataset while preventing the linkage of sensitive attributes to specific individuals.
Results and Evaluation
The tool's performance was evaluated using a location dataset from Basile et al. (2019). The Nominatim-based localizer achieved an accuracy of 0.85 for countries and 0.80 for cities, demonstrating that user-written location strings can be reliably mapped to geographical data.
At the time of publication, the tool had reached over 6,000 downloads, reflecting significant demand for standardized enrichment tools. The final output of the pipeline is a "pseudo-anonymous" dataset, where identifying information like tweet IDs and profile URLs are removed, leaving only the enriched metadata for analysis.
Limitations and Future Work
Despite its utility, the tool has limitations. Many default modules, such as Empath, are currently limited to English. The reliance on third-party services like Nominatim and Perspective API introduces dependencies on external rate limits and terms of service. Furthermore, the authors acknowledge the risks of stereotyping inherent in demographic inference and advise users to treat inferred labels as estimates.
Future developments aim to remove third-party dependencies by supporting local Nominatim indices, which would improve processing speed and eliminate external rate limits.
Summary updated