Abstract
This paper models web search behavior as an optimal stopping problem, analyzing how rational agents decide when to stop scanning search results and which link to click. By comparing a Comparison Agent (relative judgment) and a Threshold Agent (absolute quality threshold), we demonstrate that optimal search behavior is governed by a unimodal utility function balancing the benefit of finding a satisfying result against the linear cost of reviewing snippets. Our results provide closed-form solutions for optimal stopping points and show that human search patterns closely align with these rational strategies.
Introduction
When users interact with search engine results pages (SERPs), they engage in a complex decision-making process before committing to a click. While empirical studies using eye-tracking and click logs have documented how users behave, this research adopts a prescriptive approach to determine the mathematically optimal way to search. By bridging the gap between empirical observations and economic search theory, we model "pre-click" behavior as a utility maximization problem.
Theoretical Framework
The search process is modeled through a utility function defined as $Utility = Benefit - Cost$. The benefit is derived from finding a "satisfying" result, while the cost is the linear effort ($k$) required to review each additional result snippet.
We introduce two distinct rational agents:
- The Comparison Agent: This agent can judge relative quality between snippets but cannot determine if a result is truly satisfying until after clicking. It pre-determines a fixed number of results ($N$) to review and then selects the best among them.
- The Threshold Agent: This agent can predict if a result is satisfying directly from the snippet. It scans results sequentially and stops as soon as a "good enough" result is found or a maximum limit $N$ is reached.
The model assumes a Power Law of Satisfying Results, where the probability of the first satisfying result appearing at position $i$ follows $1/i^b$. Validation using AOL click logs suggests a decay exponent of $b \approx 1.9$.
Optimal Stopping Rules
A central finding is that the expected utility function for both agents is unimodal, possessing a single peak. This implies a mathematically "perfect" number of results to scan; scanning fewer results misses potential utility, while scanning more incurs excessive cognitive costs.
For the Comparison Agent, we derive a closed-form solution for the optimal number of results to review ($N_C$): $$N_C = \left\lfloor \sqrt[b]{\frac{1-p_0}{k\zeta(b)}} \right\rfloor$$ where $p_0$ represents the probability of no satisfying results and $\zeta(b)$ is the Riemann zeta function.
The optimal strategy for any rational agent is a stopping rule: scanning should cease exactly when the marginal cost of reading the next snippet exceeds the marginal expected benefit of finding a superior result.
Comparative Agent Performance
The two agents represent a trade-off between efficiency and discrimination. The Threshold Agent is more efficient, often stopping early upon finding a match. Conversely, the Comparison Agent is more discriminative, ensuring the best result within its window is found.
As illustrated in Figure 3, when the cost of reviewing results is high, the behaviors of both agents converge. However, at lower costs, the Threshold Agent typically reviews fewer results on average ($E n_T$) than the Comparison Agent ($n_C$), despite potentially setting a higher maximum limit. This efficiency stems from the Threshold Agent's ability to terminate the search as soon as a satisfying result is encountered.
Discussion and Alignment with Human Behavior
The model's predictions align closely with empirical eye-tracking data, which shows that human users rarely look past the first ten results. This suggests that human search behavior is remarkably rational given the cognitive costs involved. Furthermore, the model provides a rational explanation for "abandonment"—users stop searching when the cost of the next snippet outweighs the probability of it being satisfying.
Limitations of the current model include the assumption of static costs (ignoring varying snippet complexity) and the exclusion of post-click behavior, such as "pogo-sticking." Future work will explore non-linear costs and the impact of snippet length on the cost coefficient $k$.
Summary updated