TL;DR
This paper argues that for many social media tasks, user attributes alone can be as predictive, or even more predictive, than the content of the tweet itself. They demonstrate this by showing that models trained solely on user features can outperform state-of-the-art tweet-based models on tasks like stance detection, suggesting that some NLP tasks might be better framed as "human tasks" rather than purely document-based ones.
What Problem Does This Paper Address?
Natural Language Processing (NLP) often focuses on analyzing the content of documents (like tweets) to understand intent, sentiment, or other characteristics. However, in the context of social media, users themselves possess a wealth of information (demographics, personality, social connections, past behavior) that might be highly relevant, potentially even more so than the specific text of a single tweet. The authors address the question of how much predictive power user attributes hold independently of the tweet content, and when they are most useful, particularly for tasks that are inherently about the user rather than the immediate message. They aim to systematically evaluate this, moving beyond individual case studies to a broader understanding of user attribute utility in NLP.
Methods
The core of the paper involves building and evaluating predictive models for several NLP tasks using different feature sets:
- User Attribute-Only Models: These models use features derived solely from user profiles (name, description, location, profile picture), social network connections (followees), and inferred traits (demographics, personality, political ideology, user embeddings).
- Document-Only Models: These models use features extracted directly from the tweet text (e.g., n-grams, sentiment scores).
- Combined Models: These models integrate both user attributes and tweet features.
They employ standard machine learning classifiers like logistic regression and SVMs. A key aspect is the systematic comparison of performance across these different feature configurations for various tasks.
Datasets
The paper utilizes several established datasets for their experiments:
- Stance Detection: SemEval-2016 dataset for stance detection, which includes tweets annotated for favor, against, or neutral stances towards specific targets (atheism, climate change, feminism, Hillary Clinton, abortion).
- Sarcasm Detection: A dataset constructed by Bamman and Smith (2015), which samples tweets with and without sarcasm-indicating hashtags.
- Sentiment Analysis: SemEval-2013 dataset for message-level sentiment annotations (positive, negative, neutral).
- PP-Attachment: A dataset for prepositional phrase attachment on Twitter, combining data from Tweebank and previous work.
The authors also collected their own user attribute data by extracting information from Twitter profiles and users' past tweets (excluding retweets and task-specific tweets).
Key Insights and Ideas
- User Attributes as Primary Predictors: The most striking finding is that for tasks like stance detection, user attributes alone can achieve performance comparable to, or even exceeding, models that use tweet content. This challenges the conventional NLP focus on text.
- Trait vs. State Outcomes: The authors propose a theory that "trait-like" outcomes (stable characteristics of a user, like stance) benefit more from user attributes than "state-like" outcomes (transient states, like sentiment). This is supported by their findings, where stance and sarcasm (more trait-like) show greater gains from user features than sentiment (more state-like).
- Task-Specific Attribute Utility: Not all user attributes are equally useful. For instance, "Followees" (who a user follows) and "Inferred Factors" (demographics, personality, etc.) consistently show strong performance across tasks, while others like "Profile Picture" are less universally helpful.
- The "Human Task" Framing: The results suggest that some social media tasks might be more accurately framed as "human tasks" where understanding the individual is paramount, rather than purely document-analysis tasks.
- Complementary, Not Exclusive: While user attributes can be powerful alone, combining them with tweet features generally yields the best results, indicating that both sources of information are valuable.
Key Results
- Stance Detection: A user attribute-only model achieved a higher F1 score than the top-performing SemEval-2016 participant for stance detection, even without seeing the tweet content.
- Sarcasm and Sentiment: User attributes significantly improved performance on sarcasm detection (up to ~3.2 points gain with "Inferred Factors") and provided moderate gains for sentiment analysis.
- PP-Attachment: User attributes offered no significant predictive value for PP-attachment, reinforcing the idea that this task is purely linguistic.
- Followees and Inferred Factors: These two user attribute categories consistently showed strong performance across multiple tasks, often outperforming other individual user attribute types.
- Combined Gains: Combining user attributes with tweet features led to further improvements, with notable gains for stance detection (e.g., +18.4 points for the neutral class when combining Followees with tweet features).
Limitations and Future Work
- Noise in Predicted Attributes: The authors acknowledge that predicted user attributes (like age or gender) can be noisy. However, they argue this noise can sometimes be beneficial, as a user's predicted attribute might better capture their behavior than their actual attribute.
- Dataset Size and Diversity: While they used established datasets, the paper implicitly highlights the importance of diverse user bases and representative training samples when considering user attributes.
- Future Work: The paper suggests exploring more sophisticated methods for integrating user attributes and tweet content, such as user-factor adaptation and residualized models. They also propose that future shared tasks should consider user attribute baselines to better understand the unique contribution of document-level features. The findings also raise questions about how human annotators infer stance and whether current annotation practices might be influenced by implicit user signals.
Summary updated