The Trumpiest Trump? Identifying a Subject's Most Characteristic Tweets

Charuta Pethe, Steven SkienaEMNLP 2019 · DOI

TL;DR

This paper introduces a method to quantify how "characteristic" a tweet is of its author, going beyond simple authorship verification. They developed a model that achieves 90.37% accuracy in identifying a user's tweets and found that these "characteristic" tweets often correlate with popularity, suggesting a link between authenticity and engagement.

What Problem Does This Paper Address?

In the age of high-volume social media, understanding the essence of a user's online persona is challenging. While many tweets are generated, some are more representative of an individual's typical style and content than others. This paper tackles the problem of identifying these characteristic tweets, which has implications for generating concise profile summaries, detecting unusual activity (like account hijacking), and understanding the relationship between an author's authentic voice and their audience's engagement. The challenge lies in defining and quantifying "characteristic-ness" in a way that aligns with human intuition and can be computationally modeled.

Methods

The core of the paper is developing a "characterization score" for tweets. They frame this as a binary classification problem: distinguishing a target user's tweets (positive examples) from all other users' tweets (negative examples).

  • Authorship Verification: They experimented with five distinct computational approaches to this binary classification task, including:
  • Compression: Using Lempel-Ziv-Welch (LZW) to measure how well a tweet can be compressed by dictionaries built from the author's tweets versus others.
  • Topic Modeling: Employing Latent Dirichlet Allocation (LDA) to represent tweets by their topic distributions and using classifiers like Logistic Regression (LR) and Multi Linear Perceptron (MLP).
  • N-gram Probability: Characterizing users by their n-gram frequencies and comparing probabilities.
  • Document Embeddings: Using FastText and BERT embeddings as features for LR and MLP classifiers.
  • Token Embeddings and Sequential Modeling: This approach, using BERT embeddings fed into a Long Short-Term Memory (LSTM) network, proved to be the most effective.
  • Characterization Score: The score is derived from the classifier's confidence that a tweet belongs to the target user.
  • Iterative Sampling: To address the extreme class imbalance (billions of negative examples vs. thousands of positive), they developed an iterative sampling technique that improved test accuracy, particularly for negative examples.
  • Human Evaluation: A user study on Amazon Mechanical Turk (MTurk) was conducted to validate their model's scores against human judgment.

Datasets

The study utilized a dataset of tweets from 15 celebrities across various domains, collected from January 1, 2008, to December 1, 2018. Negative examples were sampled from Twitter's daily feed using the Twitter API. The paper does not introduce a new benchmark dataset but uses existing Twitter data.

Key Insights and Ideas

  • "Characteristic-ness" is Quantifiable: The paper demonstrates that it's possible to assign a numerical score to how representative a tweet is of an author, and this score aligns with human intuition.
  • Authenticity and Popularity Link: A surprising finding is the strong correlation between a tweet's "characteristic-ness" score and its popularity (likes, replies, retweets) for many users. This suggests that tweets that sound more "like the author" tend to be more engaging.
  • Divergent Popularity Trends: The relationship between characteristic-ness and popularity isn't uniform. Some users see higher scores correlating with more popularity (reinforcement), while others see higher scores correlating with less popularity (novelty seeking).
  • Deep Learning for Nuance: Advanced techniques like BERT embeddings combined with LSTMs were crucial for capturing the subtle linguistic nuances that define an author's style, outperforming simpler methods.
  • Handling Imbalance is Key: The extreme class imbalance in Twitter data poses a significant challenge, and their iterative sampling method was vital for achieving good performance.

Key Results

  • Best Model Accuracy: The BERT + LSTM approach achieved a mean test accuracy of 90.37% in distinguishing user tweets from non-user tweets.
  • Human Agreement: Human evaluators agreed with the model's characterization scores 70.40% of the time, with statistically significant agreement for all 15 celebrities.
  • Popularity Correlation: For 13 out of 15 celebrities, a statistically significant correlation was found between the characterization score and tweet popularity. For example, Donald Trump showed a strong positive correlation (0.64 for likes), while Ariana Grande showed a strong negative correlation (-0.74 for likes).
  • Qualitative Examples: The paper provides compelling examples of high-scoring (characteristic) and low-scoring (uncharacteristic) tweets for figures like Donald Trump, Amitabh Bachchan, Ariana Grande, and Justin Bieber, illustrating the model's ability to capture distinct authorial voices.

Limitations and Future Work

  • PR Agency Influence: The authors acknowledge that celebrity Twitter accounts can be managed by PR agencies, and their model does not distinguish between user-written and agency-written tweets. This is identified as an interesting direction for future research.
  • English-Only Tweets: The current analysis is restricted to English language tweets.
  • Punctuation as a Feature: While punctuation was removed for simplicity, the authors suggest it could prove effective as a feature in future models.
  • Temporal Effects: While Figures 3 and 4 suggest temporal effects are not the primary driver of the popularity correlation, further investigation into temporal dynamics could be beneficial.
  • Broader Applications: The authors suggest their work opens new research directions concerning virality on social media and could be applied to other forms of authorship analysis.

Summary updated

BibTeX
@inproceedings{pethe2019trumpiest,
  author = {Charuta Pethe and Steven Skiena},
  title = {{The Trumpiest Trump? Identifying a Subject's Most Characteristic Tweets}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1175},
}
Figure