Feature engineering and tree modeling for author-paper identification challenge

Jiefei Li, Xiaocong Liang, Weijie Ding, Weidong Yang, Rong Pan2013 · DOI

TL;DR

This paper tackles the challenge of accurately identifying which papers belong to a specific author, especially when author names are ambiguous and data comes from diverse, noisy sources. The authors propose a robust solution using extensive feature engineering combined with ensemble tree-based models (GBDT and RGF), achieving a high MAP@10000 score on the KDD Cup 2013 Author-Paper Identification task.

What Problem Does This Paper Address?

In the realm of academic research, efficiently finding and aggregating information about publications is crucial. Platforms like Microsoft Academic Search aim to provide this, but a significant challenge arises from author name ambiguity. Multiple authors might share the same name, or a single author might use variations of their name across publications. This ambiguity leads to noisy author profiles, where papers can be incorrectly assigned. The KDD Cup 2013 Track 1 specifically challenged participants to develop methods for accurately determining which papers in an author's profile were genuinely authored by them, given a dataset collected from various sources with inherent noise and missing information.

Methods

The core of the proposed solution lies in a two-pronged approach: extensive feature engineering and the application of ensemble tree-based models.

Feature Engineering: The authors meticulously crafted a large set of features, categorized into:

  • Paper Features: Basic attributes of the paper itself (e.g., number of authors, publication year, conference/journal ID, number of affiliations, keywords).
  • Author Features: Basic attributes of the author (e.g., total number of papers, papers published in conferences vs. journals).
  • Author-Paper Correlation Features: Features that measure the relationship between a specific author and a specific paper. This includes counts of papers published in the paper's conference/journal by the author, the frequency of the author-paper pair appearing in the data (a crucial feature), and keyword overlap.
  • Coauthor Features: Features derived from the author's coauthors, such as the total number of papers written by the author and their coauthors, and various flags indicating name or affiliation similarities between the author and their coauthors.
  • Similarity Features: Sophisticated features designed to capture name similarity between an author and their coauthors. They employ a bipartite graph matching approach using LCP (Longest Common Prefix) and Jaro-Winkler distances to measure word similarities within names, accounting for variations and order.
  • Other Features: Additional features like the count of papers in the author's predicted list and ratios of various counts.

Tree-Based Models: The authors leveraged two powerful tree-based ensemble methods:

  • Gradient Boosting Decision Tree (GBDT): A well-established boosting technique that sequentially builds decision trees, with each new tree correcting the errors of the previous ones.
  • Regularized Greedy Forest (RGF): A more advanced tree-based model that incorporates explicit regularization to prevent overfitting and uses a structured greedy search for more efficient training.

These models were then ensembled by averaging predictions from multiple models trained with different feature subsets and parameters to improve robustness and performance.

Datasets

The primary dataset used was provided by the Microsoft Academic Search for the KDD Cup 2013 Track 1 challenge. This dataset contained author profiles and associated papers, collected from various sources, leading to inherent noise and missing attributes. The authors mention splitting this data into training and validation sets, and then merging them for final training. They also describe using a private validation set for parameter tuning and feature selection.

Key Insights and Ideas

  • Feature Engineering is Paramount: The paper strongly emphasizes that the quality and quantity of engineered features are critical for success. The "CountAppeara,p" feature (how many times an author-paper pair appears in the data) was identified as the most important, suggesting that corroboration from multiple data sources significantly increases confidence in authorship.
  • Name Ambiguity is a Major Hurdle: The core problem of author name ambiguity is highlighted as the primary driver for the need for sophisticated identification methods.
  • Ensemble of Diverse Models: Combining GBDT and RGF, along with variations in feature sets and parameters, proved more effective than any single model, demonstrating the benefit of ensemble learning for robustness.
  • Sophisticated Name Similarity Matters: The novel approach to calculating name similarity using bipartite graph matching with LCP and Jaro-Winkler distances was crucial for handling complex name variations and improving coauthor-based features.
  • Post-processing Boosts Performance: A simple post-processing step, giving higher scores to papers with high "CountAppeara,p" values, further improved the MAP score, indicating that simple heuristics can complement complex models.

Key Results

The authors achieved a high MAP@10000 score of 0.98075 on the private leaderboard, ranking 5th overall. Their ensemble of GBDT and RGF models, incorporating the extensive feature engineering, outperformed individual models. For instance, Table 2 shows that their combined approach (Row 5) achieved a validation MAP of 0.982377, significantly higher than the base paper and author features alone. The use of post-processing also demonstrated a measurable improvement in MAP scores across both GBDT and RGF models (Tables 4 and 5).

Limitations and Future Work

  • Data Noise: The authors acknowledge that the dataset was collected from many different sources and had not been cleaned, leading to noise and missing attributes. While they engineered features to be robust to this noise, directly addressing the noise itself could be a future direction.
  • Short-Text Mining: The paper notes that mining short-text attributes (like paper titles and keywords) is a challenging problem, implying that more advanced natural language processing techniques could potentially yield further improvements.
  • Limited Training Data: The paper mentions that only a small percentage of authors appear in both training and validation data, suggesting a need for techniques that can effectively learn from limited data or leverage unlabeled data.
  • Ensemble Strategy: While their ensemble strategy was successful, the paper hints at exploring other ensemble methods or pairwise/listwise ranking algorithms (like LambdaMART) more deeply, as they were not fully explored due to time constraints.
  • Further Feature Exploration: The paper mentions that they incrementally added features and kept those that improved performance. This suggests that a more exhaustive search for optimal feature combinations or entirely new feature types could be beneficial.

Summary updated

Abstract

The ability to search literature and collect/aggregate metrics around publications is a central tool for modern research. Both academic and industry researchers across hundreds of scientific disciplines, from astronomy to zoology, increasingly rely on search to understand what has been published and by whom. Microsoft Academic Search is an open platform, which provides a variety of metrics and experiences for the research community, in addition to literature search. As the covering data came from many sources, the profile of an author with an ambiguous name tends to contain noise, resulting in papers that are incorrectly assigned to others. KDD Cup 2013 Track 1 challenges participants to determine which papers in an author profile were truly written by the given author.
BibTeX
@misc{li2013feature,
  author = {Jiefei Li and Xiaocong Liang and Weijie Ding and Weidong Yang and Rong Pan},
  title = {{Feature engineering and tree modeling for author-paper identification challenge}},
  year = {2013},
  doi = {10.1145/2517288.2517294},
}
Figure