On the Use of Web Search to Improve Scientific Collections

Krutarth Patel, Cornelia Caragea, Sujatha Das Gollapalli

Abstract

Traditional digital libraries rely on focused crawling from static seed lists, which often fail to capture the volatile nature of the scholarly web. We present a search-driven framework that leverages web search engines to "pull" research documents using paper titles and author names as queries. Our system employs a two-path approach: direct title search and author homepage identification via RankSVM. Experimental results demonstrate that this framework can acquire over 267,000 unique papers from 76,000 queries, significantly outperforming traditional crawling methods in both coverage and efficiency.

Introduction

Building comprehensive digital libraries like Google Scholar or CiteSeerX is a significant challenge due to the dynamic nature of academic publishing. Traditional "focused crawling" starts with known researcher homepages and follows links. However, this method is reactive; seed lists quickly become outdated as researchers change affiliations or new scholars emerge. This paper proposes a proactive strategy: using modern web search engines to discover and download scholarly articles automatically.

Methodology: The Search/Crawl/Process Framework

The core of the proposed system is a two-path framework designed to maximize document discovery while maintaining high precision.

The Search/Crawl/Process framework architecture, illustrating Path 1 (Title Search) and Path 2 (Author Search).

This path uses known paper titles as search queries. Since a title is a dense cluster of high-quality keywords, searching for one often retrieves the target paper along with other relevant documents on the same topic. The system downloads resulting PDFs and uses a Random Forest classifier—trained on 43 structural features like text density—to verify if they are research papers.

This path aims to find author homepages to perform a shallow crawl. Identifying a homepage is treated as a ranking problem rather than simple classification. Using RankSVM, the system identifies which search result is most likely the homepage among social media profiles and news articles. Once identified, a depth-2 crawl is performed to find linked PDFs.

Experimental Evaluation

The system was evaluated using a subset of CiteSeerX containing over 43,000 titles and 32,000 unique author names. A separate dataset of 42,000 DBLP homepages was used to train the RankSVM ranker.

Results and Impact

The framework proved highly efficient, obtaining 266,877 unique research papers from approximately 76,000 queries. It successfully recovered 78% of the original papers targeted.

Venn diagram comparing the document discovery of the proposed framework against traditional crawling methods and random title sets.

As shown in Figure 4, the Search/Crawl framework discovered significantly more unique papers compared to traditional breadth-first crawling in a head-to-head comparison. Specifically, the search-driven framework found 1,760 additional papers versus 1,037 for the standard crawler. The homepage identification achieved an F1-score of 0.89, while the paper classifier reached a 0.95 F1-score, indicating a robust pipeline suitable for production environments.

Conclusion

By treating web search engines as a front-end for digital libraries, this framework bypasses the limitations of static seed lists. While challenges like name ambiguity and non-paper noise (e.g., CVs or slides) remain, the results demonstrate that "pulling" documents via search is a superior strategy for maintaining up-to-date scientific collections. Future work will focus on cross-disciplinary expansion and automated topic classification.

Summary updated

BibTeX
@inproceedings{patel2020use,
  author = {Krutarth Patel and Cornelia Caragea and Sujatha Das Gollapalli},
  title = {{On the Use of Web Search to Improve Scientific Collections}},
  booktitle = {SDP @ EMNLP},
  year = {2020},
  doi = {10.18653/v1/2020.sdp-1.20},
}
Figure