Human Speed-Accuracy Tradeoffs in Search

Christina Aperjis, Bernardo A. Huberman, Fang Wu

Introduction

In the digital age, the search for information is rarely a quest for absolute perfection. Instead, it is a dynamic process of "information foraging," where users navigate a fundamental tradeoff between the accuracy (or quality) of the information they receive and the speed at which they acquire it. When a user posts a question on a community platform like Yahoo Answers or Stack Overflow, they face a recurring decision: should they wait longer for a potentially superior answer, or should they "close" the search and act on the information currently available? This paper investigates the mathematical patterns underlying human stopping behavior in these environments.

Theoretical Framework

To understand when a user decides they have enough information, the researchers proposed two complementary mathematical models:

1. The Utility Model (Speed-Quantity Tradeoff)

This model assumes that users make decisions based on a utility function $u(n)$, where $n$ is the number of answers received. The core hypothesis is that users experience diminishing marginal returns. While the first answer provides immense value, subsequent answers offer progressively smaller refinements.

Utility curves $u(n)$ demonstrating how the perceived value of information increases with the number of answers $n$, but at a decreasing rate.

As shown in the utility curves above, the gain from $n$ to $n+1$ answers decreases as $n$ grows. A user stops waiting when the expected benefit of the next answer is outweighed by the perceived cost of the time delay.

2. The Stochastic Model (Assessing Quality)

Alternatively, the value of incoming answers can be modeled as a random walk. In this stochastic view, a user continues to wait as long as the cumulative "value" of the thread remains above a certain threshold. When the value hits this threshold (the "first passage time"), the search is terminated. This model predicts that the distribution of the total number of answers should follow an Inverse Gaussian distribution, a pattern often seen in physical processes involving diffusion.

Data and Methodology

The study utilized a real-world dataset crawled from Yahoo Answers in October 2008. To ensure the analysis captured intentional human behavior rather than automated system defaults, the researchers focused on a subset of 1,536 questions. These questions met two specific criteria: the asker manually selected a "best answer" (thereby closing the thread), and they did so within 100 hours of posting. This window avoids the 4-day automatic timeout period used by the platform.

Results and Analysis

The empirical data strongly supported the theoretical models. Using a Logit model to estimate the probability of a user closing a question, the researchers analyzed three primary variables: the number of answers already received ($n$), the time between the last two answers ($l$), and the time elapsed since the most recent answer ($w$).

Statistical estimates for the Logit model, showing the impact of quantity and timing variables on the probability of closing a question.

As indicated in the table above, all three variables were statistically significant at the 0.001 level. This confirms that users do not stop randomly; they actively weigh the quantity of information and the timing of its arrival.

Key Findings

  • The "Silence" Signal: The variable $w$ (time since the last answer) proved to be a powerful predictor. As more time passes without a new contribution, users become less optimistic about receiving a better answer and more sensitive to the cost of waiting, leading them to settle for the current results.
  • Negative Correlation: The study found a clear negative correlation (-0.148) between the number of answers a question has already received and the additional time a user is willing to wait. Essentially, the more you have, the less patient you become for more.
  • Distribution Fit: The total number of answers per question followed the predicted Inverse Gaussian distribution (with parameters $\mu = 6.1$ and $\lambda = 5.8$), validating the random walk theory of information value.

Conclusion and Implications

This research demonstrates that human stopping behavior in digital search environments is remarkably predictable and follows formal economic and physical laws. By understanding these thresholds, Q&A platforms can optimize their ecosystems. For instance, if a system detects that a user is nearing their "stopping threshold," it could prioritize that question for expert answerers, ensuring the user receives a high-value contribution before they abandon the search. Future work may focus on integrating answer quality directly into these models to create a more granular understanding of the speed-accuracy tradeoff.

Summary updated

Abstract

When foraging for information, users face a tradeoff between the accuracy and value of the acquired information and the time spent collecting it, a problem which also surfaces when seeking answers to a question posed to a large community. We empirically study how people behave when facing these conflicting objectives using data from Yahoo Answers, a community driven question-and-answer site. We first study how users behave when trying to maximize the amount of acquired information while minimizing the waiting time. We find that users are willing to wait longer for an additional answer if they have received a small number of answers. We then assume that users make a sequence of decisions, deciding to wait for an additional answer as long as the quality of the current answer exceeds some threshold. The resulting probability distribution for the number of answers that a question gets is an inverse Gaussian, a fact that is validated by our data.
BibTeX
@misc{aperjis2010human,
  author = {Christina Aperjis and Bernardo A. Huberman and Fang Wu},
  title = {{Human Speed-Accuracy Tradeoffs in Search}},
  year = {2010},
  eprint = {1008.4820},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1008.4820},
  howpublished = {SSRN},
}

Loading related content...

Figure