Abstract
In Community Question Answering (CQA) platforms, matching complex questions with appropriate experts is critical for community health. We present QDEE, a framework that estimates question difficulty and user expertise by modeling the latent knowledge hierarchy. By incorporating the Expertise Gain Assumption (EGA)—the principle that users' skills evolve over time—and utilizing a "social agony" metric to resolve hierarchical conflicts, QDEE significantly improves question routing accuracy, particularly in cold-start scenarios.
Introduction
CQA sites like Stack Overflow and Yahoo! Answers face a persistent routing challenge: ensuring difficult questions reach capable experts before being buried by newer posts. Effective routing requires accurate estimation of both question difficulty and user expertise. Existing language-conscious models often fail during "cold-start" periods when text is sparse, while language-agnostic competition graphs suffer from data sparsity. QDEE addresses these issues by leveraging temporal user behavior and robust graph-based ranking.
Methodology
The QDEE framework is built upon three primary pillars designed to overcome the limitations of traditional CQA models.
Expertise Gain Assumption (EGA)
The core novelty of QDEE is the EGA, which posits that users naturally progress toward more difficult questions as they gain experience. By adding "EGA edges" between a user's chronological questions in the competition graph, the framework bridges gaps caused by sparse interaction data, creating a more connected and representative knowledge hierarchy.
Social Agony Metric
Real-world competition graphs often contain cycles that violate a strict hierarchy. Rather than discarding this data, QDEE employs a "social agony" metric. This approach penalizes edges that go from higher-ranked nodes to lower-ranked ones based on the severity of the violation. This provides a more resilient ranking than algorithms like TrueSkill, which can overfit on noisy or subjective data.
Hybrid Cold-Start Modeling
For new questions, QDEE combines textual similarity (K-Nearest Neighbors) with the asker’s recent history. By assuming that high-level experts are likely to ask difficult questions, the model maintains high accuracy even when textual context is minimal.
Experimental Setup
The framework was evaluated using two large-scale datasets:
- Stack Overflow: Eight years of data (2008–2017) across Java, Python, C#, PHP, and HTML tags.
- Yahoo! Chiebukuro: Five years of activity from a Japanese CQA site.
Ground truth for difficulty was established using "Bounties" (Stack Overflow) and "Coins" (Yahoo!), under the assumption that higher rewards correlate with greater objective difficulty.
Results and Discussion
Experimental results demonstrate that QDEE consistently outperforms baseline models across various domains.
Key findings include:
- EGA Performance Boost: Integrating EGA edges improved difficulty estimation accuracy by approximately 5% across all datasets.
- Superiority of Social Agony: The Social Agony approach outperformed TrueSkill by an average of 4%. As shown in the performance charts, the "Agony-EGA" variant consistently achieved the highest accuracy across all programming language categories compared to standard DFS and TrueSkill baselines.
- Cold-Start Resilience: The hybrid model (Text + EGA) outperformed text-only baselines by 24.9% on the Java dataset.
- Effective Routing: QDEE achieved a hit-ratio of up to 45% in identifying the best potential answerers within the top-K recommendations.
The study also identified distinct user personas, such as "Owls" (experts targeting high-difficulty questions) and "Sparrows" (users answering many easy questions to build reputation), further validating the hierarchical model.
Conclusion
QDEE provides a robust, language-agnostic framework for mapping expertise in online communities. By treating time as a proxy for skill and utilizing social agony to handle hierarchical noise, the system offers a significant advancement in CQA question routing. Future work will focus on scaling the social agony calculations for million-node graphs and addressing the "user cold-start" problem for entirely new participants.
Reference: Sun, J., Moosavi, S., Ramnath, R., & Parthasarathy, S. (2018). QDEE: Question Difficulty and Expertise Estimation in Community Question Answering Sites. Proceedings of the AAAI Conference on Artificial Intelligence.
Summary updated