Can LLMs Simulate Social Media Engagement? A Study on Action-Guided Response Generation

Zhongyi Qiu, Hanjia Lyu, Wei Xiong, Jiebo LuoarXiv 2025 · DOI · arXiv

Abstract

Simulating social media interactions involves more than just text generation; it requires modeling the specific actions users take, such as retweeting, quoting, or rewriting. This paper introduces an Action-Guided Response Generation framework to evaluate Large Language Models (LLMs) on their ability to predict and execute these engagement types. Using a dataset of 6.3 million COVID-19 vaccine-related tweets, the study compares GPT-4o-mini, o1-mini, and DeepSeek-R1 against traditional classifiers like BERT. Results show that while LLMs excel at semantic alignment and stylistic mimicry, they struggle with action prediction, often exhibiting a strong bias toward text-based responses. Furthermore, the study identifies a non-linear relationship between few-shot prompting and prediction accuracy, suggesting that substantial historical context is necessary to overcome initial performance dips.

Introduction

Social media simulation is a complex task that extends beyond mimicking the content of a post. To truly capture human behavioral dynamics, a model must understand the "vibe" of a platform and the specific ways users interact with trending topics. On platforms like X (formerly Twitter), engagement is not a monolithic "response" but a choice between distinct actions: retweeting (amplification), quoting (commentary), or rewriting (independent posting on the same topic).

Existing research often focuses on the scalability of LLM agents, but few studies address the gap in "action-guided" generation. This paper investigates whether LLMs can accurately predict these user actions and subsequently generate responses that align with the user's likely sentiment, stance, and style.

Methodology

The researchers proposed a two-stage Action-Guided Response Generation framework designed to mirror the decision-making process of a social media user.

  1. Action Prediction: Given a trending post and user metadata, the model predicts whether the user will Retweet, Quote, or Rewrite.
  2. Response Generation: If the predicted action involves text (Quote or Rewrite), the model generates the corresponding content.

The study evaluated three state-of-the-art models—GPT-4o-mini, o1-mini, and DeepSeek-R1—under various prompting configurations, ranging from zero-shot (ZS_base) to few-shot including historical user data (FS_hist).

Data and Evaluation

The study utilized a massive dataset of 6.3 million tweets concerning COVID-19 vaccinations from late 2020. From this pool, a balanced benchmark of 3,990 entries was created, featuring equal distributions of quotes, retweets, and rewrites. To evaluate the "human-ness" of the generated responses, the researchers employed Llama-3.3-70B as a semantic judge and used LIWC (Linguistic Inquiry and Word Count) to measure stylistic alignment.

Results and Analysis

The Prediction-Generation Divide

A clear divide emerged between classification and generation capabilities. A fully fine-tuned BERT model achieved the highest action prediction accuracy at 57.14%, outperforming the best zero-shot LLM (DeepSeek-R1 at 53.18%). However, LLMs proved to be superior "simulators," achieving higher semantic alignment scores for sentiment and stance when the action was correctly identified.

Zero-shot LLMs exhibited a significant bias toward predicting "Quote" actions. This is likely because LLMs are optimized for text generation and "prefer" to write a comment rather than predict a silent retweet.

Accuracy trends across different few-shot configurations, showing the performance recovery as more historical examples are provided.

Interestingly, the study found a "few-shot accuracy dip." Adding a single example (1-shot) initially decreased prediction accuracy for models like GPT-4o-mini (dropping to ~19%), likely due to overfitting on limited context. Accuracy only began to climb steadily once 5 or more examples were provided, highlighting the importance of historical action rates in predicting user behavior.

Conclusion

Action prediction remains the primary bottleneck in social media simulation. While LLMs are excellent at matching the semantic nuance of human posts, they struggle to decide how to engage without significant historical context. Future systems may benefit from Retrieval-Augmented Generation (RAG) to pull in broader user histories or hybrid architectures that combine specialized classifiers with generative LLMs.

Summary updated

Abstract

Social media enables dynamic user engagement with trending topics, and recent research has explored the potential of large language models (LLMs) for response generation. While some studies investigate LLMs as agents for simulating user behavior on social media, their focus remains on practical viability and scalability rather than a deeper understanding of how well LLM aligns with human behavior. This paper analyzes LLMs' ability to simulate social media engagement through action guided response generation, where a model first predicts a user's most likely engagement action-retweet, quote, or rewrite-towards a trending post before generating a personalized response conditioned on the predicted action. We benchmark GPT-4o-mini, O1-mini, and DeepSeek-R1 in social media engagement simulation regarding a major societal event discussed on X. Our findings reveal that zero-shot LLMs underperform BERT in action prediction, while few-shot prompting initially degrades the prediction accuracy of LLMs with limited examples. However, in response generation, few-shot LLMs achieve stronger semantic alignment with ground truth posts.
BibTeX
@misc{qiu2025can,
  author = {Zhongyi Qiu and Hanjia Lyu and Wei Xiong and Jiebo Luo},
  title = {{Can LLMs Simulate Social Media Engagement? A Study on Action-Guided Response Generation}},
  year = {2025},
  eprint = {2502.12073},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.12073},
}
Figure