Inferring latent attributes of Twitter users with label regularization

Ehsan Mohammady Ardehaly, Aron CulottaNAACL 2015

TL;DR

This paper tackles the challenge of inferring user attributes like age, ethnicity, and political orientation from social media data without relying on extensive manual labeling. They propose a "lightly supervised" approach using label regularization, which leverages noisy, heterogeneous "soft" constraints derived from external data sources like Census demographics and name trends. This method achieves accuracy competitive with fully supervised methods, even with minimal or no labeled data for training.

What Problem Does This Paper Address?

Manually annotating data for machine learning tasks, especially on dynamic platforms like social media, is a significant bottleneck. It's expensive, time-consuming, and models quickly become outdated. This is particularly true for inferring latent attributes like age, ethnicity, or political leanings, where explicit user declarations are rare. Existing supervised methods struggle with this data scarcity. The paper addresses this by exploring "lightly supervised learning," which aims to build models from readily available unlabeled data and "soft" constraints, rather than relying solely on meticulously labeled examples.

Methods

The core of the paper's contribution lies in its application of label regularization for inferring latent attributes. Instead of directly learning from labeled instances, this approach uses a pool of unlabeled data and a set of "soft constraints." These constraints are essentially educated guesses about the distribution of attributes within subsets of the unlabeled data.

The authors propose a novel way to generate these constraints by combining:

  • County-level demographics: Using U.S. Census data for age and ethnicity proportions within specific geographic areas.
  • Name trends: Leveraging Social Security Administration data on name popularity and age distributions to infer age from first names.
  • Follower/Hashtag associations: Identifying Twitter accounts or hashtags strongly associated with a particular attribute (e.g., political party) and using the follower base of these as a constraint.

A key innovation is their constraint selection algorithm. Given the sheer volume and potential noise in their generated constraints, they employ greedy and pseudo-greedy forward-selection methods to identify the most informative subset of constraints. This is crucial for improving accuracy and robustness. They use multinomial logistic regression as their classification model and optimize an objective function that minimizes the cross-entropy between the model's predicted label distribution and the expected distribution from the constraints.

Datasets

The paper utilizes both labeled and unlabeled Twitter data:

  • Labeled Datasets: Four disjoint datasets were manually annotated for validation and tuning:
  • Race/ethnicity: 770 Twitter profiles categorized into Asian, Black, Latino, White.
  • Age: 1,436 users, categorized as below 25 or above 25 years old.
  • Politician: Official Twitter accounts of U.S. Congress members (189 Democratic, 188 Republican).
  • Politician-follower: Followers of "thedemocrats" and "gop" accounts (598 likely Democrats, 632 likely Republicans). These datasets were split into 40% for tuning/training and 60% for testing.

  • Unlabeled Twitter Data: A large corpus of 18 million geolocated tweets from 2.7 million unique users was collected using the Twitter streaming API. This data was mapped to U.S. counties using Census data.

Key Insights and Ideas

  • Noisy constraints can work surprisingly well: The paper demonstrates that even with hundreds of noisy, overlapping constraints, label regularization can achieve high accuracy, often competitive with fully supervised methods, without requiring manual labels for training.
  • Constraint selection is crucial: Simply throwing all available constraints at the model doesn't yield the best results. A carefully selected subset of constraints significantly improves accuracy, especially for tasks like age classification where noise is higher.
  • Constraint types are complementary: While individual constraint types (county, name, follower) have varying strengths, combining them, especially with a good selection algorithm, leads to the best overall performance.
  • Lightly supervised learning offers a viable alternative: The approach is less reliant on labeled data, making it more adaptable to dynamic environments like social media where data and trends change rapidly.

Key Results

  • Competitive accuracy without labels: For three out of four tasks (race, political orientation, politician-follower), the proposed label regularization approach (using all constraints, "all-const") achieved accuracy within 2% of a fully supervised logistic regression baseline, despite using no user-annotated data for training.
  • Significant improvement with constraint selection: The "imp-greedy" constraint selection algorithm improved tuning set accuracy by up to 18% (absolute) compared to using all constraints. This highlights the importance of selecting informative constraints.
  • Age classification is challenging: Age classification showed a greater degradation in accuracy due to constraint noise, but constraint selection still significantly boosted performance (from 61.4% to 79.2% with "imp-greedy").
  • Follower constraints are powerful: Follower constraints generally outperformed county constraints when used in isolation.
  • High accuracy with few constraints: Constraint selection algorithms converged to high accuracy using only a small fraction (around 20) of the available constraints.

Limitations and Future Work

  • Sampling Bias: The authors acknowledge that social media users are not representative of the general population (e.g., overrepresentation of younger, urban, and minority groups). This bias in the data and constraints can lead to inaccuracies.
  • Over-tuning: Comparing tuning and testing set accuracies suggests some degree of over-tuning, particularly for age and political classification.
  • Future Work:
    • Developing methods to directly address sampling bias in constraints.
  • Exploring alternative optimization strategies.
  • Quantifying how responsive label regularization is to evolving linguistic patterns in online data.

Summary updated

BibTeX
@inproceedings{ardehaly2015inferring,
  author = {Ehsan Mohammady Ardehaly and Aron Culotta},
  title = {{Inferring latent attributes of Twitter users with label regularization}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
}
Figure