Recommending citations for academic papers

Trevor Strohman, W. Bruce Croft, David D. JensenSIGIR 2007 · OpenReview

Abstract We approach the problem of academic literature search by considering an unpublished manuscript as a query to a search system. We use the text of previous literature as well as the citation graph that connects it to find relevant related material. We evaluate our technique with manual and automatic evaluation methods, and find an order of magnitude improvement in mean average precision as compared to a text similarity baseline.

Introduction

The "information explosion" in scientific literature makes it increasingly difficult for researchers to stay current with relevant publications. Traditional search tools often rely on short keyword queries, which suffer from two major issues: vocabulary mismatch—where different terms describe the same concept—and a lack of nuance, as a few keywords cannot capture the full context of a research draft. This paper proposes a system that utilizes an entire unpublished manuscript as a query to recommend relevant academic citations, bridging the gap between textual content and structural authority.

Methodology

The proposed system employs a two-stage retrieval and ranking model designed to identify the most relevant citations for a given draft.

Candidate Selection

Initially, the system identifies the top 100 papers most textually similar to the query manuscript. This set is then expanded to include all papers cited by these 100 documents, typically resulting in a candidate pool of 1,000 to 3,000 papers.

Feature-Based Ranking

The candidate pool is ranked using a linear combination of several features, optimized via coordinate ascent for Mean Average Precision (MAP). These features include text similarity (measured via a multinomial diffusion kernel), global citation counts, and metadata such as publication year and author identity.

Table 1: Description of features used in the ranking model.

A critical component of this ranking is the Katz measure, a graph-based metric that calculates the distance between papers in a citation network by considering the number and length of paths connecting nodes. It captures the "authority" that a set of documents places on another.

Dataset

The researchers evaluated their model using a snapshot of the Rexa corpus, a large-scale collection of academic papers similar to CiteSeer.

Table 2: Summary statistics of the Rexa corpus dataset.

The dataset included approximately 1 million entries, with over 100,000 papers containing full text and 1.46 million citation links. Bibliographic information from DBLP was used to augment the data and ensure canonical references.

Key Insights

The experimental results highlighted several important findings regarding citation behavior and retrieval performance:

  • The Power of the Graph: Text similarity alone is a poor predictor of citations. While it helps identify topical relevance, the citation graph is significantly more effective at identifying authoritative or seminal works that researchers actually cite.
  • The Dominance of the Katz Measure: Among all features, the Katz measure proved to be the most vital; removing it caused performance to drop by over 50%.
  • Feature Redundancy: Many metadata features, such as paper titles or publication years, provided negligible added value when full text and graph metrics were already present. The training process even assigned a weight of zero to the "Title" feature.
  • Beyond Similarity: Researchers often cite papers for reasons beyond topical similarity, such as citing a textbook for an algorithm or a seminal paper for historical context, even if the vocabulary differs significantly from the current draft.

Results

The system was evaluated by removing bibliographies from existing papers and attempting to predict the missing citations.

  • Quantitative Performance: The baseline text-only search achieved a Mean Average Precision (MAP) of 0.0079. The full experimental model achieved a MAP of 0.1016, representing a ten-fold improvement.
  • Qualitative Analysis: In a manual case study of an operating systems research paper, the system successfully identified 14 original citations within the top 500 results. Furthermore, it surfaced highly relevant papers that the original authors had overlooked, such as papers on modular network infrastructure and process fault isolation.

Discussion and Limitations

While the system shows significant promise, several challenges remain. The current evaluation treats relevance as binary (cited or not), which does not account for degrees of relevance or space constraints in bibliographies. Additionally, the system faces a "cold start" problem for very recent papers with no incoming citations, although the text similarity component provides a partial mitigation. Future work may involve expert-led evaluations to better assess the quality of recommendations that were not originally cited by authors.

Summary updated

Abstract

We approach the problem of academic literature search by considering an unpublished manuscript as a query to a search system. We use the text of previous literature as well as the citation graph that connects it to find relevant related material. We evaluate our technique with manual and automatic evaluation methods, and find an order of magnitude improvement in mean average precision as compared to a text similarity baseline.
BibTeX
@inproceedings{strohman2007recommending,
  author = {Trevor Strohman and W. Bruce Croft and David D. Jensen},
  title = {{Recommending citations for academic papers}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2007},
  url = {https://openreview.net/forum?id=By-t88-d-r},
}
Figure