Introduction
The rapid advancement of Large Language Models (LLMs) has sparked renewed interest in the predictability of human behavior. In the digital realm, a central question persists: are our online expressions primarily a product of individual idiosyncrasies or the "echo chambers" of our social circles? This research investigates the empirical limits of predicting individual speech on social media, quantifying the information gain from personal history versus social context.
Methodology and Data Collection
To establish a baseline for predictability, the researchers utilized a massive dataset from X (formerly Twitter) collected in early 2023. The pipeline involved a sampling stage to identify 5,102 authentic users and a subsequent collection stage to gather their timelines and the timelines of their most-mentioned peers.
Predictability was measured using Negative Log-Likelihood (NLL), converted to Bits Per Character (BPC) for cross-model comparison. The study evaluated four model families: GPT-2-XL (1.5B), Llama-3 (8B), Falcon (40B), and Llama-2 (70B). Two primary approaches were tested:
- In-Context Learning: Models were prompted with 250 past tweets to predict a subsequent evaluation set.
- Finetuning: Models were "tweet-tuned" on a general corpus of 10 million tweets and then specialized on specific user or peer data.
Key Findings: Individualism vs. Social Influence
The core of the study compared the predictive power of different contexts: no context, random users (control), peer context (most mentioned accounts), and user context (the subject's own history).
The results demonstrate that individual history is the most potent predictor. Providing user context improved model certainty by 3.1σ over the no-context baseline, while peer context only offered a 2.6σ improvement. Crucially, peer context contained very little unique information that wasn't already present in the user's own history, challenging some theories regarding the efficacy of "shadow profiling" solely through social connections.
Furthermore, the study revealed that social media speech is inherently difficult to predict. Even the largest model (Llama-2-70B) with full user context only just approached the estimated entropy of the English language (~1.12 BPC). For comparison, individual tweets proved harder for LLMs to predict than financial news, highlighting the high variance and idiosyncratic nature of personal online speech.
Linguistic Nuances and Limitations
The research also identified a "dialect gap." Models performed significantly worse on tweets from Nigerian users compared to Western cohorts, suggesting that current LLMs are still heavily optimized for "Standard" English and struggle with regional variations. Additionally, a significant portion of the predictive gain from context was found to be stylistic (syntax and formatting) rather than purely semantic.
While LLMs are becoming increasingly sophisticated, this work suggests there is a fundamental "entropy floor" to human expression that remains elusive. Future research will need to explore whether these limits are a property of the models themselves or an inherent characteristic of human communication.
Summary updated