Great Question! Question Quality in Community Q&A

Sujith Ravi, Bo Pang, Vibhor Rastogi, Ravi Kumar

Introduction

In online Community Question-Answering (CQA) ecosystems like StackOverflow, the quality of questions is a primary driver of expert engagement and long-term value. However, research has traditionally focused on answer quality, often conflating question quality with popularity metrics like view counts. This paper addresses the challenge of defining an objective metric for question quality and building a system capable of predicting it at the moment of submission, before any community feedback exists.

Methodology

The researchers framed quality prediction as a binary classification task (Good vs. Bad). To avoid the "popularity trap," they defined quality as the ratio of upvotes to views ($p_i = s_i / v_i$). Their dataset consisted of approximately 66,000 questions from StackOverflow, filtered for those with at least 1,000 views to ensure stable quality signals.

The core innovation of their approach is a hierarchical topic modeling framework that captures the "aboutness" of questions at three levels:

  1. Global Topics: General subject matter (e.g., Java vs. Python).
  2. Sentence-Level Topics (STM): Local context within the question body.
  3. Global Topic Structure (GTSM): The discourse flow, modeled using a Generalized Mallows Model to capture the specific sequence of topics (e.g., problem description followed by code).

Key Insights

The study revealed several distinct characteristics that separate high-quality questions from poor ones:

  • Conciseness is a Virtue: Contrary to the belief that more detail is always better, good questions on StackOverflow tend to be slightly shorter and more focused than bad ones. Clarity and brevity are stronger indicators of quality than sheer volume of text.
Distribution of question length (tokens) for good vs. bad questions.
  • Novelty and Research Effort: By measuring similarity against existing content, the authors found that high-quality questions are generally more unique. This suggests that the best askers perform due diligence before posting, avoiding repetitive queries.
  • Long-Term Value: Analysis of search logs showed that high-quality questions attract search engine traffic at a much higher rate over several years, acting as a "long-tail" resource for the community.

Results

The researchers evaluated their models using non-linear classifiers. While a popularity-based baseline (using only view counts) achieved 61.1% accuracy, content-based models significantly outperformed it. The inclusion of structural discourse features provided the most robust results.

Incremental accuracy gains from adding local and global topic features.

The final model achieved a peak accuracy of 72.1%, demonstrating that the text and structure of a question are powerful intrinsic predictors of its eventual community value.

Conclusion

By moving beyond simple popularity and modeling the underlying discourse structure of technical inquiries, this research provides a path toward real-time quality assessment in CQA platforms. While the current model focuses on technical content, future work could explore how these markers of quality—such as conciseness and novelty—translate to more subjective or social platforms.

Summary updated

BibTeX
@inproceedings{ravi2014great,
  author = {Sujith Ravi and Bo Pang and Vibhor Rastogi and Ravi Kumar},
  title = {{Great Question! Question Quality in Community Q\&A}},
  booktitle = {ICWSM},
  year = {2014},
}
Figure