TL;DR
This paper introduces a novel method for distinguishing between individual and organizational Twitter accounts using only a single tweet and the user's profile. By leveraging a character-based convolutional neural network and a large, weakly supervised dataset, they achieve comparable accuracy to existing methods that require significantly more data, making large-scale social media analysis more efficient and accessible.
What Problem Does This Paper Address?
Researchers frequently use Twitter data for various analyses, from political science to public health. However, a crucial preprocessing step often involves understanding who is tweeting. While many analyses focus on individuals, organizations also maintain a significant presence on Twitter. Failing to distinguish between these two account types can lead to biased results, as an organization's tweets shouldn't be treated as representative of an individual's beliefs or behaviors. Existing methods for this classification, like McCorriston et al. (2015), require downloading hundreds of tweets per account and rely on relatively small, manually annotated datasets, which can be costly, time-consuming, and prone to becoming stale as accounts change or are deleted. This paper aims to address these limitations by developing a more efficient and scalable method.
Methods
The authors propose two main improvements:
- A Character-Based Convolutional Neural Network (CNN): Instead of relying on traditional word-level features or extensive tweet histories, they employ a CNN that operates directly on the characters of a user's name. This allows for learning representations of names that can be indicative of whether the account belongs to an individual or an organization.
- A Large, Weakly Supervised Dataset: They construct a significantly larger dataset than previously available by using "weak supervision" techniques. This involves leveraging Twitter lists (user-curated collections of accounts) and LinkedIn profile links as noisy but scalable sources for labeling accounts as either individual or organizational.
These methods are combined with profile-based features (e.g., follower-to-friend ratio, verification status, use of personal pronouns) to create a classification model that requires only a single tweet and the user's profile information.
Datasets
The paper introduces a newly constructed dataset for training and evaluation:
- Newly Introduced Dataset: This dataset is approximately an order of magnitude larger than previous datasets, containing around 180,000 individuals and 28,000 organizations. It was constructed using weak supervision by:
- Twitter Lists: Identifying lists with terms like "businesses" or "companies" for organizations, and "friends" or "family" for individuals.
- LinkedIn Links: Identifying users who link to their LinkedIn profiles in their Twitter bio.
- This dataset is noted as being "noisy and cheap" due to its weak supervision origins.
- Evaluation Dataset: The authors also utilize the dataset released by McCorriston et al. (2015) for evaluation, specifically for testing the performance of their models on a higher-quality, manually annotated benchmark.
Key Insights and Ideas
- Single Tweet Sufficiency: The most significant insight is that a substantial amount of information for distinguishing individuals from organizations can be gleaned from just a single tweet and the user's profile, challenging the necessity of extensive tweet histories.
- Scalability through Weak Supervision: The paper demonstrates that even "noisy" labels derived from weak supervision can be effectively used to train robust classifiers, opening doors for more scalable data collection in social media research.
- Character-Level Name Analysis: Using character-based CNNs on usernames proves to be a powerful approach, capturing subtle patterns that might be missed by word-level analysis or require more data.
- Practical Implications for Large-Scale Analysis: The developed method significantly reduces the computational and logistical burden of data collection, making it feasible to classify millions or even billions of tweets for organizational vs. individual accounts in real-time or near real-time.
Key Results
The paper presents results from two main experiments:
- Experiment 1 (Using McCorriston et al. dataset):
- The authors' n-gram and CNN models, requiring only a single tweet, achieved accuracies comparable to the Humanizr method (which uses hundreds of tweets), especially in the "full" (imbalanced) setting.
- Humanizr slightly outperformed the n-gram and CNN models, but at the cost of significantly more data.
- Experiment 2 (Using their newly constructed dataset for training, McCorriston et al. for testing):
- The CNN model showed considerable improvement, nearly matching the performance of Humanizr.
- In the "full" setting, the CNN achieved an accuracy of 94.6%, demonstrating the value of their weakly supervised dataset for training effective classifiers.
- This experiment strongly suggests that their dataset, despite its noisy origins, is valuable for classifying organizations and individuals.
Limitations and Future Work
- Simplifying Assumption: The primary limitation acknowledged is the assumption that all accounts are either individuals or organizations, ignoring other categories like bots. While they assume bot detection systems would filter these out first, their model doesn't explicitly handle them.
- Dataset Bias: The weakly supervised dataset might be biased. Accounts in popular lists or those linking to LinkedIn might be more professional or popular than a random Twitter user, potentially biasing the classifier against less popular organizational users or individuals who don't use Twitter professionally.
- Future Work:
- Investigate the correlation of their tool's predictions with bot-detection systems.
- Incorporate content features from methods like Humanizr.
- Explore ways to control for biases in their weakly-supervised dataset to improve predictions on ground-truth data.
- Extend the model to predict other non-human account types.
Summary updated