QDEE: Question Difficulty and Expertise Estimation in Community Question Answering Sites

Jiankai Sun, Sobhan Moosavi, Rajiv Ramnath, Srinivasan Parthasarathy

Abstract

In Community Question Answering (CQA) platforms, matching complex questions with appropriate experts is critical for community health. We present QDEE, a framework that estimates question difficulty and user expertise by modeling the latent knowledge hierarchy. By incorporating the Expertise Gain Assumption (EGA)—the principle that users' skills evolve over time—and utilizing a "social agony" metric to resolve hierarchical conflicts, QDEE significantly improves question routing accuracy, particularly in cold-start scenarios.

Introduction

CQA sites like Stack Overflow and Yahoo! Answers face a persistent routing challenge: ensuring difficult questions reach capable experts before being buried by newer posts. Effective routing requires accurate estimation of both question difficulty and user expertise. Existing language-conscious models often fail during "cold-start" periods when text is sparse, while language-agnostic competition graphs suffer from data sparsity. QDEE addresses these issues by leveraging temporal user behavior and robust graph-based ranking.

Methodology

The QDEE framework is built upon three primary pillars designed to overcome the limitations of traditional CQA models.

The QDEE framework architecture, illustrating the flow from user/question history to ranked resolver lists.

Expertise Gain Assumption (EGA)

The core novelty of QDEE is the EGA, which posits that users naturally progress toward more difficult questions as they gain experience. By adding "EGA edges" between a user's chronological questions in the competition graph, the framework bridges gaps caused by sparse interaction data, creating a more connected and representative knowledge hierarchy.

Social Agony Metric

Real-world competition graphs often contain cycles that violate a strict hierarchy. Rather than discarding this data, QDEE employs a "social agony" metric. This approach penalizes edges that go from higher-ranked nodes to lower-ranked ones based on the severity of the violation. This provides a more resilient ranking than algorithms like TrueSkill, which can overfit on noisy or subjective data.

Hybrid Cold-Start Modeling

For new questions, QDEE combines textual similarity (K-Nearest Neighbors) with the asker’s recent history. By assuming that high-level experts are likely to ask difficult questions, the model maintains high accuracy even when textual context is minimal.

Experimental Setup

The framework was evaluated using two large-scale datasets:

  • Stack Overflow: Eight years of data (2008–2017) across Java, Python, C#, PHP, and HTML tags.
  • Yahoo! Chiebukuro: Five years of activity from a Japanese CQA site.

Ground truth for difficulty was established using "Bounties" (Stack Overflow) and "Coins" (Yahoo!), under the assumption that higher rewards correlate with greater objective difficulty.

Results and Discussion

Experimental results demonstrate that QDEE consistently outperforms baseline models across various domains.

Comparison of accuracy across programming languages, demonstrating the performance gains of the Agony-EGA approach.

Key findings include:

  • EGA Performance Boost: Integrating EGA edges improved difficulty estimation accuracy by approximately 5% across all datasets.
  • Superiority of Social Agony: The Social Agony approach outperformed TrueSkill by an average of 4%. As shown in the performance charts, the "Agony-EGA" variant consistently achieved the highest accuracy across all programming language categories compared to standard DFS and TrueSkill baselines.
  • Cold-Start Resilience: The hybrid model (Text + EGA) outperformed text-only baselines by 24.9% on the Java dataset.
  • Effective Routing: QDEE achieved a hit-ratio of up to 45% in identifying the best potential answerers within the top-K recommendations.

The study also identified distinct user personas, such as "Owls" (experts targeting high-difficulty questions) and "Sparrows" (users answering many easy questions to build reputation), further validating the hierarchical model.

Conclusion

QDEE provides a robust, language-agnostic framework for mapping expertise in online communities. By treating time as a proxy for skill and utilizing social agony to handle hierarchical noise, the system offers a significant advancement in CQA question routing. Future work will focus on scaling the social agony calculations for million-node graphs and addressing the "user cold-start" problem for entirely new participants.

Reference: Sun, J., Moosavi, S., Ramnath, R., & Parthasarathy, S. (2018). QDEE: Question Difficulty and Expertise Estimation in Community Question Answering Sites. Proceedings of the AAAI Conference on Artificial Intelligence.

Summary updated

Abstract

In this paper, we present a framework for Question Difficulty and Expertise Estimation (QDEE) in Community Question Answering sites (CQAs) such as Yahoo! Answers and Stack Overflow, which tackles a fundamental challenge in crowdsourcing: how to appropriately route and assign questions to users with the suitable expertise. This problem domain has been the subject of much research and includes both language-agnostic as well as language conscious solutions. We bring to bear a key language-agnostic insight: that users gain expertise and therefore tend to ask as well as answer more difficult questions over time. We use this insight within the popular competition (directed) graph model to estimate question difficulty and user expertise by identifying key hierarchical structure within said model. An important and novel contribution here is the application of "social agony" to this problem domain. Difficulty levels of newly posted questions (the cold-start problem) are estimated by using our QDEE framework and additional textual features. We also propose a model to route newly posted questions to appropriate users based on the difficulty level of the question and the expertise of the user. Extensive experiments on real world CQAs such as Yahoo! Answers and Stack Overflow data demonstrate the improved efficacy of our approach over contemporary state-of-the-art models. The QDEE framework also allows us to characterize user expertise in novel ways by identifying interesting patterns and roles played by different users in such CQAs.
BibTeX
@inproceedings{sun2018qdee,
  author = {Jiankai Sun and Sobhan Moosavi and Rajiv Ramnath and Srinivasan Parthasarathy},
  title = {{QDEE: Question Difficulty and Expertise Estimation in Community Question Answering Sites}},
  booktitle = {ICWSM},
  year = {2018},
  url = {https://openreview.net/forum?id=dqfVFLG6gg},
}
Figure