TL;DR
This paper introduces STaRS, an LLM-based framework that predicts user satisfaction with AI Overviews (AIOs) by reasoning over behavioral signals like dwell time and clicks. By combining a structured reasoning strategy with a novel reinforcement learning reward-shaping mechanism, STaRS significantly outperforms traditional evaluation methods that rely solely on text quality or simple heuristics.
What Problem Does This Paper Address?
AI Overviews (AIOs)—those generated summaries at the top of search results—are becoming a staple of modern search engines. However, evaluating whether they actually satisfy users is tricky.
Current evaluation methods are mostly system-centric, focusing on whether the generated text is accurate or relevant. While useful, these don't capture the user's actual experience. On the other hand, traditional behavior-based metrics from web search (like "a long dwell time equals satisfaction") don't translate directly to AIOs. For example, if a user reads an AIO for 30 seconds and then clicks a link, does that mean the AIO was helpful and sparked curiosity, or that it was incomplete and failed to answer the question? This ambiguity makes it difficult to build automated systems that can accurately judge user satisfaction.
Methods
The authors propose STaRS (STructured Reasoning and Shaping), which treats satisfaction prediction as a reasoning task rather than a simple classification problem.
- Structured Reasoning Strategy: Instead of asking an LLM to "guess" satisfaction, STaRS prompts it to follow a step-by-step process:
- Interpret individual behavioral signals (e.g., "The user viewed the AIO for 32s").
- Assign Polarity to each signal (Positive, Negative, or Neutral).
- Synthesize these into an overall reason and a final satisfaction score (1-5).
- RL with Reward Shaping: To fine-tune the model, the authors use Group Relative Policy Optimization (GRPO). Because final satisfaction scores are "sparse" rewards (they don't tell the model why it was right or wrong), they introduced a Reward Shaping mechanism.
- They use a high-capability "teacher" LLM to generate reference interpretations.
- They use counterfactual signal selection: they remove a behavior signal and see if the predicted satisfaction score changes. If it does, that signal is "influential," and the model is specifically rewarded for interpreting it correctly.
Datasets
The study utilizes two new datasets to validate the approach:
- UserStudy: A controlled laboratory dataset containing 554 query records from 31 participants. It includes eye-tracking-style exposure data and explicit 5-point satisfaction ratings.
- SearchLog26: A large-scale industrial dataset from a commercial search engine, consisting of 6,589 query-level logs from 5.4k users. This dataset includes real-world "pop-up" feedback where users rated their satisfaction and provided text reasons for their scores.
Key Insights and Ideas
- Behavior is Context-Dependent: The same behavior (e.g., a click) can mean different things depending on the AIO content. STaRS succeeds because it looks at the interaction between the text and the behavior.
- Reasoning Improves Accuracy: Forcing the model to explain why a behavior indicates satisfaction (structured reasoning) leads to better final predictions than direct prompting.
- Not All Signals are Equal: The counterfactual selection process revealed that many user behaviors are "peripheral." By focusing the RL training on "influential" signals, the model learns more robust reasoning patterns.
- Bridging IR and GenAI: The paper successfully applies traditional Information Retrieval (IR) concepts (behavioral signals) to the new world of Generative AI (AIOs).
Key Results
STaRS consistently outperformed all baselines, including system-centric metrics (like RAGAs and TruLens) and traditional machine learning models (like GBDT and Transformers).
- On the UserStudy dataset: STaRS achieved a Spearman correlation ($\rho$) of 0.4157, significantly higher than the best non-behavioral LLM baseline (G-Eval-Sat at 0.3187).
- On the SearchLog26 dataset: The full STaRS model (with RL) reached a Pearson correlation ($r$) of 0.4638, outperforming the G-CoS baseline (0.3328) and traditional GBDT models (0.4053).
- Ablation Studies: Removing the "Structured Reasoning" component or the "Reward Shaping" led to a noticeable drop in performance, proving that both the "thinking" process and the specialized training are necessary.
Limitations and Future Work
- Data Sparsity: The model relies on explicit user feedback (ratings) for training, which is often hard to collect at scale in production environments.
- Teacher LLM Bias: The reward shaping relies on a "high-capability" LLM (like DeepSeek-V3 or GPT-4) to provide reference labels. If the teacher has biases or flawed reasoning, these could be passed down to the trained model.
- Future Directions: The authors suggest using these satisfaction predictions as a direct reward signal to optimize the AIO generation models themselves, potentially creating a self-improving search experience.
Summary updated