On the Assessment of Expertise Profiles

Richard Berendsen, Krisztian Balog, Toine Bogers, Antal van den Bosch, Maarten de Rijke

Abstract

Evaluating the effectiveness of expertise profiling systems—algorithms designed to identify the knowledge domains of individuals—is traditionally reliant on self-selected skill lists. However, these lists are often sparse and incomplete, potentially leading to inaccurate assessments of system performance. This paper investigates the impact of ground truth quality on the evaluation of expertise profiling. By conducting an assessment experiment with 239 university employees and eight profiling models, the researchers demonstrate that expert-validated, system-generated profiles provide a more comprehensive ground truth. The study reveals that system rankings are highly sensitive to the ground truth used, suggesting that current benchmarks may misrepresent the relative efficacy of profiling algorithms.

Introduction

In the field of Information Retrieval, expertise is often approached from two directions: Expert Finding (identifying the right person for a specific topic) and Expert Profiling (identifying the relevant topics for a specific person). While significant progress has been made in finding experts, the task of profiling them remains a challenge, particularly regarding evaluation.

The standard methodology for testing profiling systems involves comparing algorithmic output against a "ground truth"—typically a list of skills or categories that experts have selected for themselves in a database. However, this approach is fundamentally flawed due to human behavior; experts are often inconsistent or incomplete when self-reporting their skills. If a system correctly identifies a valid expertise that the expert simply forgot to list, traditional evaluation metrics penalize the system for a "false positive." This paper explores whether our current benchmarks provide a distorted view of algorithmic performance and proposes a more robust evaluation framework.

Methodology

To address the limitations of static ground truth, the researchers conducted a large-scale assessment experiment using the TU (Tilburg University) expert collection. This dataset, derived from the "Webwijs" public database of university researchers and teachers, includes both self-selected skills and system-generated suggestions.

Experimental Setup

  1. Model Ensemble: The researchers deployed eight distinct state-of-the-art profiling models. These models varied by language (Dutch vs. English) and their use of thesaurus-based semantic relationships to expand potential topics.
  2. Assessment Interface: Rather than relying solely on existing lists, 239 employees from Tilburg University were presented with profiles generated by the AI models.
  3. Human-in-the-Loop: The experts evaluated these suggestions on a five-point scale, providing a "judged" relevance score. They were also encouraged to add missing expertise areas and provide qualitative feedback.
  4. Comparative Benchmarking: The study compared five different Ground Truth (GT) configurations, ranging from the original self-selected lists (GT1) to the comprehensive, expert-judged assessments (GT5).

Key Insights

The Sparsity of Self-Selection

The experiment confirmed that self-selected profiles are significantly incomplete. When experts were asked to judge system-generated suggestions, the resulting profiles were 81% complete, compared to only 65% for the original self-selected lists. This indicates that experts often omit nearly a third of their relevant skills when left to their own devices, confirming the "sparsity trap" in traditional evaluation.

The Impact of Research Descriptions

A critical factor in system success was the presence of a written research description or "bio." Systems performed significantly better for experts who provided a concise summary of their work, achieving an nDCG@100 of 0.39 compared to 0.30 for those without. This suggests that structured summaries are far more valuable for profiling algorithms than raw, unstructured document clusters.

Sensitivity of System Rankings

The most significant finding for the research community is the sensitivity of the "leaderboard" to the ground truth. When moving from self-selected data to the more complete judged data, the correlation between system rankings dropped to $\tau = 0.56$. This moderate correlation implies that a system ranked as mediocre under traditional benchmarks might actually be a top performer when evaluated against a more accurate and complete representation of human expertise.

Binary vs. Graded Relevance

Interestingly, while the completeness of the list significantly altered system rankings, the granularity of the expertise (e.g., distinguishing between a "novice" and an "expert" level) did not. The relative ranking of the eight systems remained identical ($\tau = 1.00$) whether using binary (relevant/not relevant) or graded relevance scales.

Discussion and Future Work

Despite improvements in evaluation, the study highlighted several persistent challenges in automated expertise profiling:

  • Redundancy: Experts frequently noted that systems suggested multiple terms that were semantically identical, leading to cluttered and repetitive profiles.
  • Specificity and Hierarchy: There is a difficult balance between being too broad (e.g., "Social Science") and too narrow. Future systems must better navigate the hierarchical nature of knowledge.
  • Diversity: Qualitative feedback suggested that systems should prioritize a diverse range of topics to capture the multi-faceted nature of modern academic work, rather than focusing only on the most statistically frequent terms.

Conclusion

By moving beyond static, self-reported lists and incorporating expert validation into the evaluation loop, this research provides a roadmap for more accurate expertise retrieval. The findings underscore that the quality of our ground truth dictates the validity of our conclusions; to build AI that truly understands human expertise, we must first improve how we measure that understanding.

Summary updated

BibTeX
@inproceedings{berendsen2013assessment,
  author = {Richard Berendsen and Krisztian Balog and Toine Bogers and Antal van den Bosch and Maarten de Rijke},
  title = {{On the Assessment of Expertise Profiles}},
  booktitle = {DIR},
  year = {2013},
}
Figure