Beyond Action Imitation: Learning a Decision-Aware User Simulator for Online Advertising

Zipeng Chen, Jiaer Zheng, Xiangyang Xu, Xinyu Lin, Zhaobin Wang, Zhaohui Liu, Qianjin Xiang, Xiaoyu Zhao, Zhuozhen Yu, Guangshuo Wang, Daxing Chen, Junwei Pan, Zhangbin Zhu, Chengguo Yin, Hao Chen, Tat-Seng Chua, Haijie Gu, Jie JiangarXiv 2026 · arXiv

TL;DR

This paper introduces DASH, a decision-aware user simulator for online advertising that goes beyond simply mimicking user actions (like clicks) to jointly generating the underlying "thinking traces" behind those actions. By leveraging heterogeneous cross-domain interaction histories and a progressive three-stage framework (Context Engineering, Supervised Fine-Tuning, and Reinforcement Learning), DASH achieves state-of-the-art simulation fidelity and provides valuable diagnostic insights for offline recommendation evaluation.

What Problem Does This Paper Address?

Evaluating new algorithms in online advertising via live A/B testing is expensive, risky, and can degrade user experience. While Large Language Model (LLM)-based user simulators offer a promising offline alternative, existing approaches suffer from two major flaws:

  1. Single-Domain Preference Modeling: They only look at interaction histories within a single domain, missing the broader context of a user's multi-domain digital footprint.
  2. Action-Only Response Modeling: They focus solely on predicting the final action (e.g., click vs. skip). This allows models to take "shortcuts" (overfitting to specific surface features) and fails to explain why a user made a choice, stripping the simulator of diagnostic value.

Tackling these flaws introduces two core challenges: compressing massive, noisy cross-domain histories into an LLM's context window without losing critical signals (Challenge 1), and supervising "thinking traces" when no ground-truth user thoughts exist in real-world logs (Challenge 2).

Methods

The authors propose DASH (Decision-Aware Simulator with Heterogeneous context), which uses a progressive three-stage pipeline:

Figure 1: Overview of DASH, a progressive CE-SFT-RL framework for advertising user simulation. Stage 1 uses context compression and prompt optimization to leverage decision-relevant signals for effective reasoning. Stage 2 constructs token- level supervision signals, distilling quality-filtered teacher trajectories into a student model. Stage 3 further improves action fidelity and thinking quality through GRPO with a hybrid trace-level reward (i.e., action fidelity and thinking quality).
  1. Stage 1: Context Engineering (CE): To handle millions of historical tokens, DASH uses a Hierarchical Context Compression strategy. At the item level, it keeps full details for high-information actions (clicks, conversions) but retains only category metadata for low-information actions (skips). At the stream level, it prioritizes the ad stream and truncates the general content stream. It also employs a closed-loop prompt optimization process to refine a structured "Focus $\rightarrow$ Draft $\rightarrow$ Verify $\rightarrow$ Finalize" reasoning chain.
  2. Stage 2: Supervised Fine-Tuning (SFT): To give a smaller, deployable student model (Qwen3.5-35B) basic reasoning capabilities, DASH distills thinking trajectories from a powerful teacher model (Qwen3.5-397B). It curates both "easy" samples (where the teacher naturally predicts the correct action) and "hard" samples (where the teacher is forced to regenerate a reasoning trace conditioned on the true action).
  3. Stage 3: Reinforcement Learning (RL): Using Group Relative Policy Optimization (GRPO), DASH aligns the student model using a hybrid reward function. This combines a sparse action fidelity reward with a dense, rubric-based thinking quality reward that scores the generated text across three dimensions: Form, Content, and Logic.

Datasets

The model was trained and evaluated on a large-scale, real-world cross-domain advertising dataset collected from the Tencent platform.

  • The dataset includes user profiles, ad creative profiles, ad request contexts, and a 30-day interaction history.
  • The interaction history spans five heterogeneous content domains: Apps, Games, Videos, Articles, and Search.
  • On average, each ad request is accompanied by a massive raw context of roughly 3.73 million tokens before compression.

Key Insights and Ideas

  • Logic Over Content: An intriguing finding was that the logical coherence of a user's simulated thinking trace correlates much more strongly with accurate action prediction than factual content density alone. Models that generated highly plausible content but lacked causal consistency performed poorly at predicting final actions.
  • The "Lost-in-the-Middle" Remedy: Standard LLMs often drop critical historical signals when buried deep inside a long prompt. DASH's explicit "Focus" stage acts as an information retriever, pulling out and timestamping strong negative signals (like explicit negative feedback) before the reasoning process begins.
  • RL as a Shortcut Preventer: Relying purely on SFT data limits the model to the teacher's coverage. Introducing trace-level RL rewards prevents the model from taking superficial shortcuts, ensuring it doesn't fabricate unobserved user needs to justify a predicted click.

Key Results

DASH significantly outperformed both its base model and massive general-purpose LLMs (including 1T parameter models like Kimi-K2.5) across action prediction and thinking quality metrics.

  • Action Fidelity: DASH (Small-scale SFT + RL) achieved a Weighted F1 score of 62.15%, outperforming the teacher model (Qwen3.5-397B at 60.23%) and the baseline student model (55.18%).
  • Thinking Quality: DASH achieved the highest overall logic score (92.62) and overall thinking quality score (92.01).
  • Distribution Alignment: As shown below, the optimized prompt structure drastically reduced the vanilla model's tendency to over-predict clicks and under-predict conversions, aligning closely with real-world action frequencies.
Figure 5: Ratio of predicted to real action frequency under Vanilla and Optimized prompts.

Limitations and Future Work

While highly effective, the authors note that deploying LLM-based simulators at scale introduces dual-use risks, such as the potential for ad networks to over-optimize aggressive ranking or targeting strategies against simulated users. To mitigate this, they restrict DASH to evaluative and diagnostic use cases.

Future work focuses on integrating DASH directly into production infrastructure for three key applications: offline A/B test pre-screening, pre-testing ad creatives before they are served to real users, and optimizing the balance between platform monetization and user experience.

Summary updated

Abstract

Recent advances in LLM-based user simulation have shown promise for offline evaluation of recommendation and advertising systems. However, existing simulators typically infer user preferences from single-domain interaction histories and are primarily optimized to reproduce observable actions such as clicks. Consequently, they capture only a partial view of user preferences, while action-only prediction easily induces model shortcuts and limits both the fidelity and diagnostic value of simulation. To address these challenges, we propose DASH, a decision-aware user simulator that jointly generates thinking traces and predicts behavioral actions from heterogeneous cross-domain histories. DASH first introduces a Context Engineering stage that folds heterogeneous cross-domain histories into decision-relevant context, together with prompt optimization for effective reasoning over the folded context. To train a user simulator, DASH distills thinking trajectories from strong LLMs as SFT data, and further tailors a rubric-based reward model that evaluates thinking traces along form, content, and logic for RL training. Combined with the action reward, these signals jointly improve action prediction and thinking quality. Extensive experiments on real-world Tencent advertising data spanning five heterogeneous content domains demonstrate the effectiveness, efficiency, fidelity, and diagnostic value of DASH.
BibTeX
@misc{chen2026beyond,
  author = {Zipeng Chen and Jiaer Zheng and Xiangyang Xu and Xinyu Lin and Zhaobin Wang and Zhaohui Liu and Qianjin Xiang and Xiaoyu Zhao and Zhuozhen Yu and Guangshuo Wang and Daxing Chen and Junwei Pan and Zhangbin Zhu and Chengguo Yin and Hao Chen and Tat-Seng Chua and Haijie Gu and Jie Jiang},
  title = {{Beyond Action Imitation: Learning a Decision-Aware User Simulator for Online Advertising}},
  year = {2026},
  eprint = {2607.26893},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.26893},
}
Figure