Abstract
Simulating social media interactions involves more than just text generation; it requires modeling the specific actions users take, such as retweeting, quoting, or rewriting. This paper introduces an Action-Guided Response Generation framework to evaluate Large Language Models (LLMs) on their ability to predict and execute these engagement types. Using a dataset of 6.3 million COVID-19 vaccine-related tweets, the study compares GPT-4o-mini, o1-mini, and DeepSeek-R1 against traditional classifiers like BERT. Results show that while LLMs excel at semantic alignment and stylistic mimicry, they struggle with action prediction, often exhibiting a strong bias toward text-based responses. Furthermore, the study identifies a non-linear relationship between few-shot prompting and prediction accuracy, suggesting that substantial historical context is necessary to overcome initial performance dips.
Introduction
Social media simulation is a complex task that extends beyond mimicking the content of a post. To truly capture human behavioral dynamics, a model must understand the "vibe" of a platform and the specific ways users interact with trending topics. On platforms like X (formerly Twitter), engagement is not a monolithic "response" but a choice between distinct actions: retweeting (amplification), quoting (commentary), or rewriting (independent posting on the same topic).
Existing research often focuses on the scalability of LLM agents, but few studies address the gap in "action-guided" generation. This paper investigates whether LLMs can accurately predict these user actions and subsequently generate responses that align with the user's likely sentiment, stance, and style.
Methodology
The researchers proposed a two-stage Action-Guided Response Generation framework designed to mirror the decision-making process of a social media user.
- Action Prediction: Given a trending post and user metadata, the model predicts whether the user will Retweet, Quote, or Rewrite.
- Response Generation: If the predicted action involves text (Quote or Rewrite), the model generates the corresponding content.
The study evaluated three state-of-the-art models—GPT-4o-mini, o1-mini, and DeepSeek-R1—under various prompting configurations, ranging from zero-shot (ZS_base) to few-shot including historical user data (FS_hist).
Data and Evaluation
The study utilized a massive dataset of 6.3 million tweets concerning COVID-19 vaccinations from late 2020. From this pool, a balanced benchmark of 3,990 entries was created, featuring equal distributions of quotes, retweets, and rewrites. To evaluate the "human-ness" of the generated responses, the researchers employed Llama-3.3-70B as a semantic judge and used LIWC (Linguistic Inquiry and Word Count) to measure stylistic alignment.
Results and Analysis
The Prediction-Generation Divide
A clear divide emerged between classification and generation capabilities. A fully fine-tuned BERT model achieved the highest action prediction accuracy at 57.14%, outperforming the best zero-shot LLM (DeepSeek-R1 at 53.18%). However, LLMs proved to be superior "simulators," achieving higher semantic alignment scores for sentiment and stance when the action was correctly identified.
The "Quote" Bias and Few-Shot Trends
Zero-shot LLMs exhibited a significant bias toward predicting "Quote" actions. This is likely because LLMs are optimized for text generation and "prefer" to write a comment rather than predict a silent retweet.
Interestingly, the study found a "few-shot accuracy dip." Adding a single example (1-shot) initially decreased prediction accuracy for models like GPT-4o-mini (dropping to ~19%), likely due to overfitting on limited context. Accuracy only began to climb steadily once 5 or more examples were provided, highlighting the importance of historical action rates in predicting user behavior.
Conclusion
Action prediction remains the primary bottleneck in social media simulation. While LLMs are excellent at matching the semantic nuance of human posts, they struggle to decide how to engage without significant historical context. Future systems may benefit from Retrieval-Augmented Generation (RAG) to pull in broader user histories or hybrid architectures that combine specialized classifiers with generative LLMs.
Summary updated