Usefulness of click-through data in expert search

Craig Macdonald, Ryen W. White

Introduction

Finding the right person for a specific task is a persistent challenge in large organizations. Traditional "expert finding" systems typically rely on content evidence—analyzing document authorship and mentions. However, authorship does not always guarantee expertise. This paper, presented at SIGIR 2009, explores a "wisdom of the crowd" approach by leveraging intranet search logs. The authors demonstrate that user behavior, specifically click-through data, serves as a powerful implicit endorsement of expertise.

Methodology

The researchers investigated whether behavioral evidence from search logs could complement traditional text-based retrieval. They compared two primary click-based ranking strategies against a baseline expert search engine:

  • ClickVotes: Ranks experts based on the number of unique documents associated with them that were clicked for a specific query.
  • ClickCount: Ranks experts based on the total volume of clicks across all their associated documents.
  • Hybrid Approach: A linear combination of the baseline engine and the ClickCount scores ($w_1=1, w_2=0.1$).

To map documents to people, the authors utilized an internal API to identify authors who remained active members of the organization.

Data Collection and Ground Truth

To evaluate these methods, the authors developed a new test collection using intranet logs from a global organization with over 150,000 employees.

Summary of the test collection statistics

The researchers identified 20 popular queries with clear expertise needs. Ground truth was established by directly emailing 340 candidate experts to verify their own expertise, resulting in 199 high-quality responses (a 58.5% response rate).

Results and Analysis

The study evaluated performance using Mean Reciprocal Rank (MRR), Precision at 3 (P@3), and Mean Average Precision (MAP).

Performance comparison of different expert search techniques

The results indicate that while click data alone (ClickCount) is less effective than a sophisticated baseline engine, it provides a highly complementary signal. The hybrid approach achieved a MAP of 0.5014, representing a statistically significant improvement of over 10% compared to the baseline's 0.4479. Furthermore, ClickCount outperformed ClickVotes, suggesting that the intensity of clicks on high-quality documents is a stronger indicator of expertise than the breadth of documents authored. The hybrid model also maintained a high P@3 of 0.4833, meaning nearly half of the top three suggested experts were confirmed experts.

Conclusion

This research demonstrates that intranet click-through logs are a valuable, underutilized resource for expert search. By treating clicks as implicit peer reviews, organizations can better identify true subject matter experts. While the "cold start" problem remains for rare queries, the integration of behavioral signals significantly enhances the accuracy of internal talent discovery.

Summary updated

Abstract

The task in expert finding is to identify members of an organisation with relevant expertise on a given topic. Typically, an expert search engine uses evidence from the authors of on-topic documents found in the organisation's intranet by search engines. The search result click-through behaviour of many intranet search engine users provides an additional source of evidence to identify topically-relevant documents, and via document authorship, experts. In this poster, we assess the usefulness of click-through log data for expert finding. We find that ranking authors based solely on the clicks their documents receive is reasonably effective at correctly identifying relevant experts. Moreover, we show that this evidence can successfully be integrated with an existing expert search engine to increase its retrieval effectiveness.
BibTeX
@inproceedings{macdonald2009usefulness,
  author = {Craig Macdonald and Ryen W. White},
  title = {{Usefulness of click-through data in expert search}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2009},
  doi = {10.1145/1571941.1572143},
}
Figure