Learned Citation Contexts for Uncited Papers in Retrieve-then-Rerank Search

Search systems that must be fast work in two steps. A cheap first-stage retriever pulls a few dozen to a few hundred candidates from millions of documents. An expensive second-stage model, such as a cross-encoder (a transformer that reads the query and a document together), then reorders those candidates. Anything the first stage misses is lost for good. A well-established way to help the first stage is to index each document together with what other people wrote when they linked to or cited it: anchor text on the web, and citation sentences in science. In scientific retrieval this "referral" text roughly doubles the share of relevant papers found in the top 10. That evidence has one weakness. A newly published paper has nobody citing it, so it gets no help. Adding referrals to well-cited papers may then actively push uncited ones out of a small candidate pool.

We propose to measure this effect directly and to fix it. The fix is a model that learns, from the organic citation sentences written about older papers, to write the sentences a community would plausibly use to cite a new paper. We then index those synthetic citation contexts for documents that have no real ones. The claim is concrete. Measured by how many citations a paper has, the recall gain from referral indexing goes overwhelmingly to the already-cited head of the corpus, and in small candidate windows uncited papers are crowded out. Learned citation contexts recover the uncited papers' share without diluting the head. As a result, a downstream re-ranker reaches its best accuracy with fewer candidates.

Core Claim

Referral Augmentation for Zero-Shot Information Retrieval supplies the method we build on. It appends citing passages to a paper's index entry, or pools their embeddings, and raises BM25 Recall@10 on ACL from 0.265 to 0.505. It also beats query-generation expansion (DocT5Query) by 37 points absolute. The same paper names the limitation that motivates this proposal: referral augmentation cannot help documents with no citations. Web data show how large that population is. TREC-10 Web Track Experiments at MSRA found that about 28% of pages had no incoming anchor text and 75% had fewer than ten words of it.

Referral augmentation was evaluated only as a single-stage method, with corpus-wide recall averages. Nobody has asked who gains and who loses inside a fixed candidate budget. Our hypothesis is that within a top-$k$ window, augmented well-cited papers take slots from uncited relevant papers. If so, average recall rises while recall on recent papers falls, and recent papers are precisely what a citation-recommendation user most needs.

The proposed remedy differs from generative document expansion such as the pointer-generator query expansion in IDST at TREC 2019 Deep Learning Track. That kind of expansion predicts search queries. Ours predicts citation sentences, the same kind of text that made referral augmentation work. We supervise it on real referrals and apply it only where real referrals are missing or thin.

Research Program

The first component is a diagnostic. We take scientific corpora with citation graphs and publication dates. We freeze the citation graph at a cutoff date, so that papers published after the cutoff truly have no referrals, and we evaluate on queries drawn from papers written after the cutoff. For each method we report candidate recall at $k$ from 10 to 1000, broken down by the target's in-degree at the cutoff: zero, one to four, and five or more citations. We also report a crowding statistic. This is the fraction of uncited relevant papers that make the top-$k$ without augmentation but are displaced from it once the rest of the corpus is augmented. If crowding is negligible, the proposal narrows to the re-ranking questions below. We will report that outcome honestly rather than argue around it.

The second component is the citation-context generator. It is a sequence-to-sequence model that takes a paper's title and abstract and outputs several short sentences in the style of in-text citations. We train it on (cited paper, citing sentence) pairs, using only pairs where both papers predate the cutoff, to avoid leakage. At indexing time, each paper with fewer than $m$ real referrals receives synthetic ones, enough to bring its total up to $m$. We tune $m$ on a development split.

Because concatenating many referrals inflates term counts and lengths in BM25, we also test a capped variant. It indexes a fixed-size sample of referrals per paper, real ones first. The capped variant separates the "more text" effect from the "right kind of text" effect.

The third component is the cascade. A SciBERT cross-encoder of the kind used in Navigation-Based Candidate Expansion and Pretrained Language Models for Citation Recommendation re-ranks the first-stage pool. We vary two things: the pool size, and whether the re-ranker reads the abstract alone or the abstract plus referral text, real or synthetic. Following the saturation analysis in CoRT: Complementary Rankings from Transformers, which found a BERT re-ranker peaking at 64 candidates on a hybrid pool versus 256 on BM25 alone, we report the smallest $k$ at which end-to-end MRR comes within 0.5 points of its maximum. Separately, we report end-to-end quality on uncited targets.

Baselines

The do-nothing reference is plain BM25 over title and abstract, followed by the same cross-encoder. It is the floor every pipeline must clear at every $k$. It also serves as the non-augmented reference for measuring crowding, because under it uncited papers compete on equal terms.

The strongest existing first-stage method is referral augmentation itself, Referral Augmentation for Zero-Shot Information Retrieval. We run it both as BM25 with concatenated referrals and as dense mean-pooling, over the identical frozen graph. Its reported ACL figure is BM25+RAR Recall@10 = 0.505, which we expect to reproduce approximately before the time split is imposed. Under the time split, our method beats this baseline only if two conditions both hold. Recall@$k$ on uncited targets must rise significantly at $k \le 100$, and recall on well-cited targets must not drop by more than one point. Each method gets the same number of BM25 parameter settings tuned on the same development queries.

The strongest pipeline competitor is navigation-based candidate expansion from Navigation-Based Candidate Expansion and Pretrained Language Models for Citation Recommendation. It adds papers cited by the initial BM25 candidates to the pool and reports Recall@1000 rising from 42% to 62% on the Open Research Corpus. We reimplement it on our corpora with the same re-ranker and the same total candidate budget. Graph expansion structurally cannot reach uncited papers through cited-by links, so we expect it to win on the well-cited head and lose on the uncited tail. Combining the two methods is a natural final configuration.

The generative-expansion baseline is DocT5Query-style query prediction, as in IDST at TREC 2019 Deep Learning Track. It uses the same model size, the same training compute, and the same number of generated strings per paper, but it is trained to output queries rather than citation sentences. This is the ablation that isolates our key ingredient: the citation-sentence target distribution. A further ablation applies synthetic contexts to every paper rather than only the under-cited ones. This tests whether gating by in-degree matters, or whether synthetic text is simply useful everywhere.

Why not just…?

Why not just make the candidate pool bigger? Raising $k$ does eventually admit crowded-out papers. The cost is that re-ranker work grows linearly with $k$. CoRT: Complementary Rankings from Transformers shows that a better first stage cut the needed pool fourfold, and Beyond Sequential Reranking: Reranker-Guided Search Improves Reasoning Intensive Retrieval works under a budget of 100 documents precisely because LLM re-rankers cannot afford more. Our saturation-$k$ measurement answers this directly. If the baseline must go to $k=1000$ to match our method at $k=100$, the larger pool has a tenfold re-ranking cost.

Why not just use DocT5Query on the uncited papers? This is the real threat. It is cheap and coverage is universal. The evidence against it is indirect. Referrals beat DocT5Query by 37 points absolute on ACL in Referral Augmentation for Zero-Shot Information Retrieval, but that comparison pits real human text against synthetic text. It does not test synthetic citations against synthetic queries. Our matched ablation, with the same model and budget and only the output target changed, settles the question. If query generation matches citation generation on uncited papers, the contribution shrinks to the crowding diagnosis plus in-degree gating, and we will say so.

Why not just use a dense retriever, which handles vocabulary mismatch without extra text? Dense encoders still gain heavily from referrals: SimCSE Recall@10 goes from 0.160 to 0.355 in the referral-augmentation paper, so they do not close the gap that referrals fill. They are also brittle under input noise; Dealing with Typos for BERT-based Passage Retrieval and Ranking reports MRR@10 dropping from 0.296 to 0.141 under query typos. We nonetheless include dense mean-pooling with real and synthetic referrals as a first-stage arm, so the comparison is shown rather than assumed.

Why not just boost uncited papers with a recency or structural prior? Combining Structural Information and the Use of Priors in Mixed Named-Page and Homepage Finding shows that a URL prior applied naively to a flat model drops MRR from 0.612 to 0.315. A single scalar boost raises every recent paper indiscriminately, rather than the ones whose content matches the query. We test a tuned additive recency prior as a cheap arm. If it recovers most of the uncited-paper recall, synthetic contexts are not worth their cost.

Proposed Experiments

The primary benchmarks are ACL, ArXiv, and RefSeer from the referral-augmentation study. We add the Open Research Corpus and the DBLP and PubMed time splits used by Navigation-Based Candidate Expansion and Pretrained Language Models for Citation Recommendation. The prior summaries do not transcribe numbers for ArXiv, RefSeer, DBLP, or PubMed, so we report no reference figures for those datasets and establish our own baselines. For a web-side replication we would use TREC .GOV anchor text, where Finding "Abstract Fields" of Web Pages and Query Specific Retrieval reports a 49.97% MAP gain from anchors. There we would simulate cold start by hiding anchors for a random subset of pages, a stratum that TREC-10 Web Track Experiments at MSRA suggests is large in real crawls.

The first decisive check is whether referral augmentation lowers top-100 recall on uncited targets relative to plain BM25. That would confirm crowding exists. The second is whether citation-context generation beats matched query generation on uncited targets at $k \le 100$. The third is whether the full pipeline reaches the re-ranker's best MRR at a smaller pool than referral augmentation alone, while improving end-to-end MRR on recent papers. We treat failure of the first check as reducing the work to a re-ranking-input study. Failure of the second reduces it to an in-degree-gated expansion study, which is still useful but narrower.