Question Difficulty Estimation in Community Question Answering Services

Jing Liu, Quan Wang, Chin-Yew Lin, Hsiao-Wuen HonEMNLP 2013 · OpenReview

Abstract

In this paper, we address the problem of estimating question difficulty in community question answering services. We propose a competition-based model for estimating question difficulty by leveraging pairwise comparisons between questions and users. Our experimental results show that our model significantly outperforms a PageRank-based approach. Most importantly, our analysis shows that the text of question descriptions reflects the question difficulty. This implies the possibility of predicting question difficulty from the text of question descriptions.

Introduction

In massive Community Question Answering (CQA) ecosystems like Stack Overflow, "one size fits all" question routing is inefficient. Experts are often under time constraints and should not be bogged down by trivial questions, while beginners might be intimidated by research-level problems. Automatically determining if a question is "hard" or "easy" is vital for routing questions to the right experts, designing fair incentive systems (e.g., awarding more reputation for harder tasks), and optimizing user engagement.

Prior attempts to estimate difficulty, such as those using PageRank, often assumed that difficulty was the primary factor in determining a "best answer." However, this research argues that the outcome of a question thread depends on both the difficulty of the task and the expertise of the competitors.

Methodology: The Competition Model

The core innovation of this work is framing question answering as a competition. A question $q$ is treated as a "pseudo-user" ($u_q$). When a user $u_a$ asks a question and a user $u_b$ provides the best answer, several "matches" are recorded:

  • The question $u_q$ "beats" the asker $u_a$ (as the asker could not solve it).
  • The best answerer $u_b$ "beats" the question $u_q$ (as they successfully solved it).
  • The best answerer $u_b$ "beats" the asker $u_a$ and all other non-best answerers.

To resolve these wins and losses into difficulty scores, the authors utilize TrueSkill, a Bayesian skill rating system originally developed for matchmaking in competitive gaming. Unlike PageRank, TrueSkill accounts for uncertainty and the relative expertise of opponents. For instance, if a high-expertise user struggles with a question, the question's difficulty rating rises significantly.

Datasets and Validation

The study utilized three primary datasets from the Stack Exchange network (2008–2012):

  • SO/CPP: 122,012 C++ programming questions from Stack Overflow.
  • SO/Math: 51,174 mathematics questions from Stack Overflow.
  • MO: 27,333 research-level mathematics questions from MathOverflow.

To validate the model, graduate students manually annotated 300 pairs of questions to create a "ground truth" for difficulty comparisons.

Key Results and Analysis

Model Performance

The competition-based model achieved 66.5% accuracy on C++ questions and 71.8% on Math questions when compared to human expert judgments. It significantly outperformed the PageRank-based baseline, which performed near the level of random guessing (~50%). This confirms that accounting for user expertise is essential when measuring task difficulty.

Linguistic Markers of Difficulty

A key finding is the strong correlation between question difficulty and the language used in descriptions. In the mathematics datasets, tags like homework were strong indicators of easy questions, whereas set-theory or general-topology indicated high difficulty, reflecting the natural progression of a student's learning process.

Word clouds illustrating the shift in topics from easy to hard questions in mathematics.

In the C++ dataset, basic terms like list and array were associated with the easiest questions, while advanced concepts like virtual functions and gcc compiler issues were linked to much higher difficulty scores.

Cross-Community Calibration

By linking users who participate in both MathOverflow (high-level) and Stack Overflow Math (general-level), the model successfully calibrated the scales across different communities. The analysis proved that MathOverflow questions are statistically much harder than those on Stack Overflow, with their difficulty distributions peaking at significantly higher scores.

Difficulty distribution across MathOverflow (MO) and Stack Overflow Math (SO/Math).

Conclusion and Future Work

While the competition-based model is highly effective, it currently suffers from a "cold start" problem: it requires answers and votes to estimate difficulty. Future work aims to build predictive models that can estimate difficulty purely from the text of a question at the moment it is posted. Additionally, while the model excels in technical fields, further research is needed to determine its applicability in non-technical or subjective domains where the distinction between "expert" and "non-expert" may be less clear.

Summary updated

Abstract

In this paper, we address the problem of estimating question difficulty in community question answering services. We propose a competition-based model for estimating question difficulty by leveraging pairwise comparisons between questions and users. Our experimental results show that our model significantly outperforms a PageRank-based approach. Most importantly, our analysis shows that the text of question descriptions reflects the question difficulty. This implies the possibility of predicting question difficulty from the text of question descriptions.
BibTeX
@inproceedings{liu2013question,
  author = {Jing Liu and Quan Wang and Chin-Yew Lin and Hsiao-Wuen Hon},
  title = {{Question Difficulty Estimation in Community Question Answering Services}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2013},
  url = {https://openreview.net/forum?id=Sk404ffdbB},
}
Figure