Weakly Learning to Match Experts in Online Community

Yujie Qian, Jie Tang, Kan Wu

Abstract

In online question-and-answer (QA) platforms, identifying the right expert involves more than just matching expertise to a topic; it requires predicting an expert's willingness to respond. This paper introduces the Weakly supervised Factor Graph (WeakFG) model, which balances expertise matching with participation likelihood by incorporating social identity theory. Validated on QA-Expert and Paper-Reviewer datasets, WeakFG demonstrates a significant performance improvement of 1.5% to 10.7% in Mean Average Precision (MAP) over state-of-the-art baselines.

Introduction

Traditional expert finding systems treat the problem as an information retrieval task, focusing solely on the relevance of a user's knowledge to a query. However, in real-world communities like Quora or academic peer review, a significant portion of invitations are declined or ignored. This "willingness gap" leads to inefficiencies where high-quality questions remain unanswered. The challenge lies in trading off the matching degree of expertise with the probability of a positive response, often under conditions of extreme data sparsity.

Methodology

The proposed WeakFG model addresses these challenges by integrating two primary components into a unified framework:

  • Expertise Matching (Local Factors): To move beyond simple keyword matching, the model utilizes word embeddings—specifically Word Mover’s Distance (WMD) and Document Vector (D2V)—to represent questions and expert profiles in a continuous semantic space.
  • Acceptance Probability (Correlation Factors): Drawing from social identity theory, the model assumes that users with similar social identities (e.g., shared affiliations or interests) exhibit correlated behaviors. If one member of a social group declines an invitation, others in that group are statistically more likely to do the same.
The WeakFG model architecture, illustrating the integration of local expertise factors and social correlation factors.

The model employs a factor graph to propagate known response labels to unlabeled nodes via these social correlations, using Loopy Belief Propagation (LBP) for inference and parameter estimation.

Experimental Evaluation

The model was validated using two distinct datasets and a real-world deployment:

  1. QA-Expert: 182 questions and 599 users from a ByteCup competition.
  2. Paper-Reviewer: 935 submissions and 440 PC members from a major Computer Science conference, where bidding behavior served as a proxy for willingness.
  3. Online System: A real-world deployment for journal editors across 10 international journals to recommend reviewers for 540 papers.

Results and Discussion

WeakFG consistently outperformed baselines such as RankSVM, LDA, and standard Language Models (LM). Specifically, the model achieved a MAP improvement of up to 10.7%.

Comprehensive results: (a) Ablation study showing the impact of different components; (b) Performance stability across training ratios; (c) Decline rates in the online journal system; (d) Comparison against the LM baseline.

Ablation studies confirmed that removing the social correlation component leads to a noticeable drop in performance, highlighting the importance of social context. Furthermore, the online evaluation demonstrated that the model effectively reduces the declination rate for journal reviewer invitations compared to expertise-only matching, proving its practical utility in high-stakes environments.

Conclusion

By explicitly modeling both expertise and willingness through a weakly supervised factor graph, WeakFG provides a more holistic approach to expert recommendation. The integration of social identity theory allows the model to overcome data sparsity issues common in online communities. Future work will focus on automating the discovery of social correlations and improving the model's scalability for larger, more dynamic online platforms.

Summary updated

Abstract

In online question-and-answer (QA) websites like Quora, one central issue is to find (invite) users who are able to provide answers to a given question and at the same time would be unlikely to say "no" to the invitation. The challenge is how to trade off the matching degree between users' expertise and the question topic, and the likelihood of positive response from the invited users. In this paper, we formally formulate the problem and develop a weakly supervised factor graph (WeakFG) model to address the problem. The model explicitly captures expertise matching degree between questions and users. To model the likelihood that an invited user is willing to answer a specific question, we incorporate a set of correlations based on social identity theory into the WeakFG model. We use two different genres of datasets: QA-Expert and Paper-Reviewer, to validate the proposed model. Our experimental results show that the proposed model can significantly outperform (+1.5-10.7% by MAP) the state-of-the-art algorithms for matching users (experts) with community questions. We have also developed an online system to further demonstrate the advantages of the proposed method.
BibTeX
@inproceedings{qian2018weakly,
  author = {Yujie Qian and Jie Tang and Kan Wu},
  title = {{Weakly Learning to Match Experts in Online Community}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.24963/ijcai.2018/534},
}
Figure