From User Behaviors to Satisfaction Judgments: Reasoning about User Satisfaction with AI Overviews

Xingzhu Wang, Yitong Xu, Yue Li, Daiting Shi, Jingzhou He, Jiaxin MaoarXiv 2026 · arXiv

TL;DR

This paper introduces STaRS, an LLM-based framework that predicts user satisfaction with AI Overviews (AIOs) by reasoning over behavioral signals like dwell time and clicks. By combining a structured reasoning strategy with a novel reinforcement learning reward-shaping mechanism, STaRS significantly outperforms traditional evaluation methods that rely solely on text quality or simple heuristics.

What Problem Does This Paper Address?

AI Overviews (AIOs)—those generated summaries at the top of search results—are becoming a staple of modern search engines. However, evaluating whether they actually satisfy users is tricky.

Current evaluation methods are mostly system-centric, focusing on whether the generated text is accurate or relevant. While useful, these don't capture the user's actual experience. On the other hand, traditional behavior-based metrics from web search (like "a long dwell time equals satisfaction") don't translate directly to AIOs. For example, if a user reads an AIO for 30 seconds and then clicks a link, does that mean the AIO was helpful and sparked curiosity, or that it was incomplete and failed to answer the question? This ambiguity makes it difficult to build automated systems that can accurately judge user satisfaction.

Figure 1: Illustration of (a) a SERP containing an AI overview (with a satisfaction rating card) and (b) inputs and output for our AI overview satisfaction prediction task.

Methods

The authors propose STaRS (STructured Reasoning and Shaping), which treats satisfaction prediction as a reasoning task rather than a simple classification problem.

  1. Structured Reasoning Strategy: Instead of asking an LLM to "guess" satisfaction, STaRS prompts it to follow a step-by-step process:
  2. Interpret individual behavioral signals (e.g., "The user viewed the AIO for 32s").
  3. Assign Polarity to each signal (Positive, Negative, or Neutral).
  4. Synthesize these into an overall reason and a final satisfaction score (1-5).
  5. RL with Reward Shaping: To fine-tune the model, the authors use Group Relative Policy Optimization (GRPO). Because final satisfaction scores are "sparse" rewards (they don't tell the model why it was right or wrong), they introduced a Reward Shaping mechanism.
  6. They use a high-capability "teacher" LLM to generate reference interpretations.
  7. They use counterfactual signal selection: they remove a behavior signal and see if the predicted satisfaction score changes. If it does, that signal is "influential," and the model is specifically rewarded for interpreting it correctly.
Figure 2: The proposed structured reasoning strategy. Given the query 𝑞, AI overview 𝑎, behavior signals 𝐵, and context information 𝐶(red), the LLM is prompted to generate a structured reasoning trace (blue).

Datasets

The study utilizes two new datasets to validate the approach:

  • UserStudy: A controlled laboratory dataset containing 554 query records from 31 participants. It includes eye-tracking-style exposure data and explicit 5-point satisfaction ratings.
  • SearchLog26: A large-scale industrial dataset from a commercial search engine, consisting of 6,589 query-level logs from 5.4k users. This dataset includes real-world "pop-up" feedback where users rated their satisfaction and provided text reasons for their scores.

Key Insights and Ideas

  • Behavior is Context-Dependent: The same behavior (e.g., a click) can mean different things depending on the AIO content. STaRS succeeds because it looks at the interaction between the text and the behavior.
  • Reasoning Improves Accuracy: Forcing the model to explain why a behavior indicates satisfaction (structured reasoning) leads to better final predictions than direct prompting.
  • Not All Signals are Equal: The counterfactual selection process revealed that many user behaviors are "peripheral." By focusing the RL training on "influential" signals, the model learns more robust reasoning patterns.
  • Bridging IR and GenAI: The paper successfully applies traditional Information Retrieval (IR) concepts (behavioral signals) to the new world of Generative AI (AIOs).

Key Results

STaRS consistently outperformed all baselines, including system-centric metrics (like RAGAs and TruLens) and traditional machine learning models (like GBDT and Transformers).

  • On the UserStudy dataset: STaRS achieved a Spearman correlation ($\rho$) of 0.4157, significantly higher than the best non-behavioral LLM baseline (G-Eval-Sat at 0.3187).
  • On the SearchLog26 dataset: The full STaRS model (with RL) reached a Pearson correlation ($r$) of 0.4638, outperforming the G-CoS baseline (0.3328) and traditional GBDT models (0.4053).
  • Ablation Studies: Removing the "Structured Reasoning" component or the "Reward Shaping" led to a noticeable drop in performance, proving that both the "thinking" process and the specialized training are necessary.
Table 2: Overall performance on the UserStudy dataset.

Limitations and Future Work

  • Data Sparsity: The model relies on explicit user feedback (ratings) for training, which is often hard to collect at scale in production environments.
  • Teacher LLM Bias: The reward shaping relies on a "high-capability" LLM (like DeepSeek-V3 or GPT-4) to provide reference labels. If the teacher has biases or flawed reasoning, these could be passed down to the trained model.
  • Future Directions: The authors suggest using these satisfaction predictions as a direct reward signal to optimize the AIO generation models themselves, potentially creating a self-improving search experience.

Summary updated

Abstract

The growing presence of AI overviews in everyday search makes their effective evaluation essential. Response quality has been the primary focus of existing evaluation research, with only a small number of studies examining the user perspective. These evaluations rely on queries and response content, leaving the information conveyed by user behavior insufficiently utilized. This work investigates satisfaction prediction for AI overviews from a user-centered perspective and incorporates behavioral signals into the prediction process. Applying existing behavior-based satisfaction prediction methods from traditional information retrieval to this setting, however, presents limitations. We address these limitations with STaRS, an LLM-based approach designed specifically for predicting satisfaction with AI overviews. At its core is a structured reasoning procedure: individual behavioral signals are interpreted first, and their implications then inform the prediction of user satisfaction. Reinforcement learning further optimizes this procedure, while a reward shaping mechanism alleviates reward sparsity during training. Evaluation on two real-world datasets demonstrates that STaRS outperforms the comparison methods in satisfaction prediction, providing search providers with a practical means of accurately estimating users' satisfaction with AI overviews.
BibTeX
@misc{wang2026from,
  author = {Xingzhu Wang and Yitong Xu and Yue Li and Daiting Shi and Jingzhou He and Jiaxin Mao},
  title = {{From User Behaviors to Satisfaction Judgments: Reasoning about User Satisfaction with AI Overviews}},
  year = {2026},
  eprint = {2611.10715},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10715},
}
Figure