Harvesting Collective Intelligence: Temporal Behavior in Yahoo Answers

Christina Aperjis, Bernardo A. Huberman, Fang Wu

Abstract

When harvesting collective intelligence from online communities, users face a fundamental conflict: maximizing the accuracy and volume of information while minimizing the time spent collecting it. This paper empirically investigates this tradeoff using data from Yahoo Answers. We employ two complementary approaches: a utility-based model focusing on the quantity of information and waiting costs, and a stochastic model focusing on the perceived quality of individual answers. Our findings reveal that users experience decreasing marginal returns for additional answers and that the distribution of answer counts follows an inverse Gaussian pattern, suggesting a threshold-based stopping process.

The Stopping Problem in Social Q&A

Seeking help from the "crowd" involves a speed-accuracy tradeoff. While more answers generally lead to more comprehensive conclusions, waiting for those answers incurs a temporal cost. This study seeks to identify the "stopping rule" users employ: at what point does the cost of waiting for the next answer outweigh its potential benefit? Understanding this behavior is essential for platforms to optimize question visibility and maximize value for the community.

Modeling User Behavior

The researchers propose two distinct frameworks to explain how users decide to close a question and select a best answer:

1. The Utility Model (Quantity-Focused)

This model treats the number of answers as a proxy for information accuracy. It assumes a "myopic decision rule" where a user closes a question if the expected cost of waiting for one more answer exceeds the incremental benefit. Using a logit model (logistic regression), the authors estimate the probability of a question closing based on the current answer count, the time between the last two answers, and the total time elapsed since the most recent answer.

2. The Stochastic Model (Quality-Focused)

This approach focuses on the perceived value of individual answers rather than just the passage of time. The sequence of answer values is modeled as a random walk. A user continues to wait as long as the value of the current answer remains above a specific threshold. Mathematically, this model predicts that the total number of answers per question should follow an inverse Gaussian distribution, which is a heavy-tailed, Zipf-like distribution.

Empirical Findings

The study analyzed two datasets from Yahoo Answers (Dataset A with ~81,000 questions and Dataset B with ~1,500 questions), focusing on threads where the author manually selected a "best answer" within 100 hours.

Decreasing Marginal Utility and Waiting Costs

The data strongly supports the hypothesis of decreasing marginal returns. The first few answers are perceived as the most valuable; as the total count grows, the benefit of waiting for "just one more" drops significantly.

Mean elapsed time until the question is closed vs. the total number of answers received

As shown in the figure above, there is a clear negative correlation between the total number of answers a question receives and the time the user is willing to wait after the last answer arrives. For instance, users wait over 50 hours on average after the first answer, but this drops to less than 35 hours after the fifth answer. This suggests that the "waiting cost" is dynamic and influenced by the perceived likelihood of receiving a timely, high-quality response.

Distribution of Answers

The stochastic model's prediction regarding answer distribution was validated by the data. The frequency of answers per question follows a heavy-tailed distribution, characteristic of many social systems.

Frequency distribution of the number of answers per question on a log-log scale

Dataset B showed a remarkably close fit to the inverse Gaussian distribution (mean $\mu=6.1$, scale $\lambda=5.8$). This fit supports the theory that users perform a form of sequential sampling, stopping once they hit a satisfaction or boredom threshold based on answer quality.

Conclusion and Implications

The study demonstrates that human behavior in collective intelligence systems is predictable through utility and stochastic modeling. The probability of a user closing a question increases with the number of answers already received and the time elapsed since the last response.

These insights offer practical applications for platform optimization. Q&A sites could use these utility functions to prioritize questions currently in the "high marginal utility" phase (those with very few answers) on their front pages. By strategically directing the crowd's attention, platforms can maximize the total social surplus and the overall quality of harvested intelligence.

Summary updated

Abstract

When harvesting collective intelligence, a user wishes to maximize the accuracy and value of the acquired information without spending too much time collecting it. We empirically study how people behave when facing these conflicting objectives using data from Yahoo Answers, a community driven question-and-answer site. We take two complementary approaches. We first study how users behave when trying to maximize the amount of the acquired information, while minimizing the waiting time. We identify and quantify how question authors at Yahoo Answers trade off the number of answers they receive and the cost of waiting. We find that users are willing to wait more to obtain an additional answer when they have only received a small number of answers; this implies decreasing marginal returns in the amount of collected information. We also estimate the user's utility function from the data. Our second approach focuses on how users assess the qualities of the individual answers without explicitly considering the cost of waiting. We assume that users make a sequence of decisions, deciding to wait for an additional answer as long as the quality of the current answer exceeds some threshold. Under this model, the probability distribution for the number of answers that a question gets is an inverse Gaussian, which is a Zipf-like distribution. We use the data to validate this conclusion.
BibTeX
@misc{aperjis2010harvesting,
  author = {Christina Aperjis and Bernardo A. Huberman and Fang Wu},
  title = {{Harvesting Collective Intelligence: Temporal Behavior in Yahoo Answers}},
  year = {2010},
  eprint = {1001.2320},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1001.2320},
}
Figure