Author Homepage Discovery in CiteSeerX

Krutarth Patel, Cornelia Caragea, Doina Caragea, C. Lee Giles

Abstract

Maintaining an up-to-date index of researcher homepages is a significant challenge for digital libraries. This paper proposes a "search-then-classify" framework that leverages author names and research paper titles as search queries to discover homepages across the web. By employing a dual-branch Convolutional Neural Network (CNN) and a "Teacher-Student" self-training approach, the system identified over 18,000 new homepages, significantly expanding the coverage of scholarly databases like DBLP and CiteSeerX.

Introduction

Scholarly search engines like CiteSeerX rely on researcher homepages as "seed URLs" to crawl and index new scientific documents. However, these URLs are highly volatile; approximately 42% of homepages in a DBLP subset were found to be outdated within three years due to institutional moves or affiliation changes. Traditional broad crawling within university domains is often inefficient and misses researchers in industry labs. There is a critical need for an automated, scalable method to discover homepages "in the wild."

Methodology

Search-Driven Discovery

The authors introduce a two-stage pipeline. First, candidate URLs are generated via the Bing search API using two query paths:

  1. Author Name Queries: Standard searches for researcher names.
  2. Paper Title Queries: A novel approach using research paper titles. These titles are rich in semantic keywords and often lead directly to the personal sites of authors or co-authors.

Deep Learning Classification

To verify if a candidate URL is indeed a homepage, the authors developed a dual-branch CNN classifier. This model processes two distinct information streams: the URL string itself and the HTML content of the page.

The dual-branch CNN architecture combines URL-based and content-based features for high-accuracy classification.
  • URL Branch: Analyzes the structure and keywords (e.g., ~, homepage) within the URL string.
  • Content Branch: Analyzes the first 1,000 words of the page's HTML content.

Self-Training (Teacher-Student)

To address the scarcity of labeled data, a semi-supervised "Teacher-Student" approach was implemented. A "Teacher" model, trained on a small labeled set, generates pseudo-labels for a large unlabeled dataset. This augmented data is then used to train a more robust "Student" model.

The Teacher-Student self-training framework allows the model to leverage large amounts of unlabeled data to improve performance.

Results and Insights

The combined CNN model achieved an F1-score of 0.92 and an accuracy of 92.35%, outperforming traditional baselines. In a large-scale deployment using 10,000 titles and ~14,000 authors from CiteSeerX, the system discovered 20,593 unique homepages, 18,711 of which were previously unknown to the initial DBLP-derived dataset.

Key insights from the study include:

  • Navigational Power of Titles: Paper titles are highly effective for uncovering niche research communities and co-author networks.
  • The Non-EDU Landscape: Approximately 51% of discovered homepages were outside the .edu domain, emphasizing the limitations of domain-restricted crawling.
  • Effectiveness of Self-Training: Self-training improved the F1-score from 0.78 to 0.84 when using only 1% of the labeled training data.

Conclusion

The proposed framework demonstrates a powerful method for automating the discovery of researcher homepages at scale. While the system is highly accurate, future work will focus on periodic maintenance to handle the dynamic nature of the web and refining the classifier to better distinguish personal homepages from similar academic pages like course sites or lab member lists.

Reference: Patel, K., Caragea, C., Caragea, D., & Giles, C. L. (2021). Author Homepage Discovery in CiteSeerX. Proceedings of the AAAI Conference on Artificial Intelligence, 35(17), 15146-15154.

Summary updated

Abstract

Scholarly digital libraries provide access to scientific publications and comprise useful resources for researchers. CiteSeerX is one such digital library search engine that provides access to more than 10 million academic documents. We propose a novel search-driven approach to build and maintain a large collection of homepages that can be used as seed URLs in any digital library including CiteSeerX to crawl scientific documents. Precisely, we integrate Web search and classification in a unified approach to discover new homepages: first, we use publicly-available author names and research paper titles as queries to a Web search engine to find relevant content, and then we identify the correct homepages from the search results using a powerful deep learning classifier based on Convolutional Neural Networks. Moreover, we use Self-Training in order to reduce the labeling effort and to utilize the unlabeled data to train the efficient researcher homepage classifier. Our experiments on a large scale dataset highlight the effectiveness of our approach, and position Web search as an effective method for acquiring authors' homepages. We show the development and deployment of the proposed approach in CiteSeerX and the maintenance requirements.
BibTeX
@inproceedings{patel2021author,
  author = {Krutarth Patel and Cornelia Caragea and Doina Caragea and C. Lee Giles},
  title = {{Author Homepage Discovery in CiteSeerX}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i17.17778},
}
Figure