A framework for constructing a huge name disambiguation dataset: algorithms, visualization and human collaboration

Zhuoyue Xiao, Yutao Zhang, Bo Chen, Xiaozhao Liu, Jie TangarXiv 2020 · arXiv

Introduction

Author Name Disambiguation (AND) is a critical challenge in maintaining the integrity of academic databases like Google Scholar and AMiner. The core problem lies in correctly attributing publications to distinct individuals who share identical names (e.g., "Wei Wang"). While machine learning offers a potential solution, the field has long been hindered by a lack of high-quality, large-scale training data. Existing benchmarks are often too small to capture the complexity of real-world scenarios where a single name might correspond to thousands of distinct authors.

To address this, we introduce WhoisWho, the world's largest manually-labeled name disambiguation dataset, and the collaborative AI-assisted framework used to build it.

The Collaborative Annotation Framework

Manually labeling hundreds of thousands of documents is logistically impossible due to the quadratic growth of pairwise comparisons. Our framework overcomes this bottleneck by integrating automated AI refinement with a structured human-in-the-loop workflow.

The complete AI-assisted annotation pipeline, from candidate set generation to final result.

1. Relation Graph Construction and GNN Refinement

The process begins by constructing similarity graphs based on document attributes such as co-authors and venues. We employ an Edge Graph Neural Network (EGNN) with adaptive graph pruning to predict the likelihood of two documents sharing an author. By feeding edge features directly into the GNN decoder, we significantly boost performance; the topological structure of the similarity graph is as informative as the node attributes themselves.

Initial relation graph construction where documents are linked based on similarity and grouped by potential authors.

2. Sub-clustering and Visualization

To streamline the human task, the system uses the Speaker-Listener Label Propagation Algorithm (SLPA) to group highly similar documents into sub-clusters. Instead of evaluating individual papers, annotators interact with these clusters through a custom UI. The interface visualizes document clusters as interactive circles, highlighting potential links and shared features to facilitate rapid decision-making.

3. Structured Annotation Workflow

To ensure data quality and prevent cascading errors like "over-merging," we enforce a four-step workflow:

  • Cleaning: Removing noise and misassigned papers from initial clusters.
  • Verifying: Cross-checking assignments through multiple annotators and majority voting.
  • Adding: Assigning previously unassigned papers to the correct clusters.
  • Merging: Combining clusters that represent the same physical author.

The WhoisWho Dataset

The resulting WhoisWho dataset represents a significant leap in scale over previous benchmarks. Sampled from the AMiner database, it contains 399,255 documents belonging to 45,187 distinct authors, covering 421 highly ambiguous names. For comparison, the CiteSeerX benchmark contains only 8,466 documents across 14 names. This scale allows for the training of much more robust and generalizable disambiguation models.

Key Results and Insights

The proposed framework and model set a new state-of-the-art baseline for the AND task:

  • Model Performance: Our GNN model achieved a Micro Average F1-score of 83.71% and a Macro Average F1-score of 80.39% on the WhoisWho benchmark, outperforming the previous AMiner baseline by over 22%.
  • Human Efficiency: The system enabled annotators to process 600-700 documents per hour, allowing a team to label 100,000 documents in just 300 man-hours.
  • Reliability: The "excluding ratio" (the rate of correction during verification) remained under 5%, indicating high initial annotation accuracy.

Conclusion

The WhoisWho project demonstrates that human-AI synergy is essential for scaling complex data labeling tasks. By combining topological graph learning with structured human workflows and interactive visualization, we have created a resource that can drive the next generation of name disambiguation algorithms. While a performance gap still exists between current models and human accuracy, this dataset provides the necessary foundation for future research.

Summary updated

BibTeX
@misc{xiao2020framework,
  author = {Zhuoyue Xiao and Yutao Zhang and Bo Chen and Xiaozhao Liu and Jie Tang},
  title = {{A framework for constructing a huge name disambiguation dataset: algorithms, visualization and human collaboration}},
  year = {2020},
  eprint = {2007.02086},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.02086},
}
Figure