Introduction
Finding the right person for a specific task is a persistent challenge in large organizations. Traditional "expert finding" systems typically rely on content evidence—analyzing document authorship and mentions. However, authorship does not always guarantee expertise. This paper, presented at SIGIR 2009, explores a "wisdom of the crowd" approach by leveraging intranet search logs. The authors demonstrate that user behavior, specifically click-through data, serves as a powerful implicit endorsement of expertise.
Methodology
The researchers investigated whether behavioral evidence from search logs could complement traditional text-based retrieval. They compared two primary click-based ranking strategies against a baseline expert search engine:
- ClickVotes: Ranks experts based on the number of unique documents associated with them that were clicked for a specific query.
- ClickCount: Ranks experts based on the total volume of clicks across all their associated documents.
- Hybrid Approach: A linear combination of the baseline engine and the
ClickCountscores ($w_1=1, w_2=0.1$).
To map documents to people, the authors utilized an internal API to identify authors who remained active members of the organization.
Data Collection and Ground Truth
To evaluate these methods, the authors developed a new test collection using intranet logs from a global organization with over 150,000 employees.
The researchers identified 20 popular queries with clear expertise needs. Ground truth was established by directly emailing 340 candidate experts to verify their own expertise, resulting in 199 high-quality responses (a 58.5% response rate).
Results and Analysis
The study evaluated performance using Mean Reciprocal Rank (MRR), Precision at 3 (P@3), and Mean Average Precision (MAP).
The results indicate that while click data alone (ClickCount) is less effective than a sophisticated baseline engine, it provides a highly complementary signal. The hybrid approach achieved a MAP of 0.5014, representing a statistically significant improvement of over 10% compared to the baseline's 0.4479. Furthermore, ClickCount outperformed ClickVotes, suggesting that the intensity of clicks on high-quality documents is a stronger indicator of expertise than the breadth of documents authored. The hybrid model also maintained a high P@3 of 0.4833, meaning nearly half of the top three suggested experts were confirmed experts.
Conclusion
This research demonstrates that intranet click-through logs are a valuable, underutilized resource for expert search. By treating clicks as implicit peer reviews, organizations can better identify true subject matter experts. While the "cold start" problem remains for rare queries, the integration of behavioral signals significantly enhances the accuracy of internal talent discovery.
Summary updated