Abstract
In this paper, we address the problem of estimating question difficulty in community question answering services. We propose a competition-based model for estimating question difficulty by leveraging pairwise comparisons between questions and users. Our experimental results show that our model significantly outperforms a PageRank-based approach. Most importantly, our analysis shows that the text of question descriptions reflects the question difficulty. This implies the possibility of predicting question difficulty from the text of question descriptions.
Introduction
In massive Community Question Answering (CQA) ecosystems like Stack Overflow, "one size fits all" question routing is inefficient. Experts are often under time constraints and should not be bogged down by trivial questions, while beginners might be intimidated by research-level problems. Automatically determining if a question is "hard" or "easy" is vital for routing questions to the right experts, designing fair incentive systems (e.g., awarding more reputation for harder tasks), and optimizing user engagement.
Prior attempts to estimate difficulty, such as those using PageRank, often assumed that difficulty was the primary factor in determining a "best answer." However, this research argues that the outcome of a question thread depends on both the difficulty of the task and the expertise of the competitors.
Methodology: The Competition Model
The core innovation of this work is framing question answering as a competition. A question $q$ is treated as a "pseudo-user" ($u_q$). When a user $u_a$ asks a question and a user $u_b$ provides the best answer, several "matches" are recorded:
- The question $u_q$ "beats" the asker $u_a$ (as the asker could not solve it).
- The best answerer $u_b$ "beats" the question $u_q$ (as they successfully solved it).
- The best answerer $u_b$ "beats" the asker $u_a$ and all other non-best answerers.
To resolve these wins and losses into difficulty scores, the authors utilize TrueSkill, a Bayesian skill rating system originally developed for matchmaking in competitive gaming. Unlike PageRank, TrueSkill accounts for uncertainty and the relative expertise of opponents. For instance, if a high-expertise user struggles with a question, the question's difficulty rating rises significantly.
Datasets and Validation
The study utilized three primary datasets from the Stack Exchange network (2008–2012):
- SO/CPP: 122,012 C++ programming questions from Stack Overflow.
- SO/Math: 51,174 mathematics questions from Stack Overflow.
- MO: 27,333 research-level mathematics questions from MathOverflow.
To validate the model, graduate students manually annotated 300 pairs of questions to create a "ground truth" for difficulty comparisons.
Key Results and Analysis
Model Performance
The competition-based model achieved 66.5% accuracy on C++ questions and 71.8% on Math questions when compared to human expert judgments. It significantly outperformed the PageRank-based baseline, which performed near the level of random guessing (~50%). This confirms that accounting for user expertise is essential when measuring task difficulty.
Linguistic Markers of Difficulty
A key finding is the strong correlation between question difficulty and the language used in descriptions. In the mathematics datasets, tags like homework were strong indicators of easy questions, whereas set-theory or general-topology indicated high difficulty, reflecting the natural progression of a student's learning process.
In the C++ dataset, basic terms like list and array were associated with the easiest questions, while advanced concepts like virtual functions and gcc compiler issues were linked to much higher difficulty scores.
Cross-Community Calibration
By linking users who participate in both MathOverflow (high-level) and Stack Overflow Math (general-level), the model successfully calibrated the scales across different communities. The analysis proved that MathOverflow questions are statistically much harder than those on Stack Overflow, with their difficulty distributions peaking at significantly higher scores.
Conclusion and Future Work
While the competition-based model is highly effective, it currently suffers from a "cold start" problem: it requires answers and votes to estimate difficulty. Future work aims to build predictive models that can estimate difficulty purely from the text of a question at the moment it is posted. Additionally, while the model excels in technical fields, further research is needed to determine its applicability in non-technical or subjective domains where the distinction between "expert" and "non-expert" may be less clear.
Summary updated