Stepping-Stone Reading Paths: Ordering Papers to Close the Novice–Expert Vocabulary Gap

A newcomer who asks "how do computers figure out what a pronoun refers to" will not find the papers an expert would assign, because those papers say "coreference resolution," "mention-pair models," and "span ranking." Search engines treat this vocabulary mismatch as a retrieval failure to be fixed in one jump, usually by rewriting the query into expert terms. We propose treating it as a sequencing problem instead. A novice cannot read an expert paper directly. They can read a paper whose language is close to their own, which introduces a few new terms, then a paper that assumes those terms and introduces a few more. A good reading path is a chain of stepping stones, where each paper introduces only a small amount of unexplained vocabulary given what came before it. The chain retrieves expert papers that the raw query could never reach, and it puts them in a readable order.

Our claim is that one global optimization can do both jobs. Retrieval and ordering become a single search for the cheapest path from the novice's words to the expert literature. We cast this as a prize-collecting version of the asymmetric traveling salesman problem (a routing problem where the cost of going from A to B differs from B to A, and the route may skip stops at a penalty). Candidate papers are the stops. The cost of an edge from paper $i$ to paper $j$ is the vocabulary in $j$ that neither $i$ nor the query explains. Papers that cover the same concept are grouped so that the route visits at most one per group. If this works, a system can recover expert reading lists from lay queries better than query rewriting or LLM list generation. The test is whether it does so most clearly where the vocabulary gap is widest and where the topic is too recent for an LLM to have memorized.

Core Claim

Existing work splits the problem in two. ACL-rlg builds 85 expert reading lists from ACL tutorials and documents a large gap between novice queries and expert terminology. That finding motivates this proposal. However, ACL-rlg scores systems only as flat ranking (Recall@20, MRR@20, NDCG@20) and proposes no way to order papers. RePaGer does optimize a reading structure: a node-and-edge-weighted Steiner tree over a 6-million-paper citation graph. But it seeds that tree from raw keyword queries sent to Google Scholar, so a lay query yields lay seeds, and its output is a tree rather than an ordered path. BerTSP shows that ordering items as an asymmetric traveling salesman problem avoids three failures of topological sort: it breaks on cycles, it throws away confidence scores by thresholding, and its output depends on input order. BerTSP only orders a fixed, clean set of sentences. AMR-to-text as a TSP supplies the device of clusters with "pick one realization per group," solved with OR-Tools. We reuse that device for interchangeable papers.

What is new is the edge cost. Vocabulary mismatch becomes the asymmetric transition cost, so closing the gap gradually is exactly what the solver minimizes. Let $q$ be the query, let $T(x)$ be the set of salient technical terms in the title and abstract of paper $x$ (or the query words when $x=q$), and let $P(i \to j)$ be a learned probability that $i$ should be read before $j$. The edge cost is

$$c(i,j) = \alpha \frac{|T(j) \setminus (T(i) \cup T(q))|}{|T(j)|} + (1-\alpha)\,(1 - P(i \to j)),$$

and the solver chooses a path $\pi = (q, v_1, \dots, v_k)$ with $k \le L$ that maximizes $\sum_t r(v_t) - \lambda \sum_t c(v_{t-1}, v_t)$. Here $r(v)$ is a topical-relevance prize, $\lambda$ trades relevance against readability, $\alpha$ weights vocabulary gap against learned precedence, and at most one paper is visited per concept cluster.

The cost is pairwise, so it only credits terms explained by the immediately preceding paper. The true "has this term been explained yet" condition depends on the whole prefix. We treat this as a known approximation. We also test the 3-place generalization from Computing Locally Coherent Discourses, which reduces the problem to a generalized TSP solvable by branch-and-cut but scales much worse (20-sentence instances took nearly two minutes there).

Research Program

The system has three parts.

The first is a candidate pool built by expansion. Start with dense and BM25 retrieval on the raw query. Take the top papers, add their term sets $T$ to form second-hop queries, and repeat for two or three hops. This yields a pool of roughly 100 to 200 papers from S2ORC that spans lay to expert registers. Papers enter the pool with a relevance prize $r$ discounted by the hop at which they were found.

The second part learns precedence $P(i \to j)$ with a cross-encoder in the BerTSP style. It is trained self-supervised on S2ORC citation direction, where a cited earlier paper precedes the citing one, so no reading-list labels are consumed. We validate this component on prerequisite benchmarks before using it. Clusters come from embedding similarity with a threshold tuned on SurveyBank.

The third part is the solver. We use OR-Tools routing with disjunctions, which natively expresses "visit at most one per cluster, or skip at a penalty." For pools of 50 or fewer papers we also solve an exact ILP by branch-and-cut to measure the heuristic's optimality gap. Because the objective is linear in the edge costs, blackbox solver differentiation applies directly. That method perturbs the solver's inputs along the loss gradient and re-solves to get a surrogate gradient. We use it as a secondary experiment to fine-tune the precedence encoder on path-level loss instead of pairwise loss, and we adopt it only if it beats pairwise training.

Baselines

The naive reference point is direct retrieval on the raw novice query (BM25 and a dense retriever over the same S2ORC index), truncated to $L$ papers and sorted by publication year. It shows how far the vocabulary gap leaves plain search. ACL-rlg reports that academic search engines had low Recall@20, but no reported number was found on Lacuna for this.

The strongest existing competitor for recall is GPT-4o prompted to produce an ordered reading list, as benchmarked in ACL-rlg. There it achieved the highest MRR@20 of all systems, though no reported numeric result was found on Lacuna for GPT-4o on ACL-rlg. We run it with the same prompt protocol and discard references not found in the index, since ACL-rlg found about 3% hallucinated. We report its scores separately on pre-2023 and 2023+ tutorials.

The strongest existing structured method is RePaGer's Steiner tree. It reports beating Google Scholar, AMiner, PageRank, and SciBERT on SurveyBank F1 and precision, but no reported numeric result was found on Lacuna for RePaGer. We run it on our citation graph twice: once with its native keyword seeds, and once with our expanded pool as seeds. The second run separates the value of our pool from the value of path optimization.

The competitor we consider most threatening is query rewriting followed by chronological sorting. An LLM, and separately a RaFe-style rewriter trained with reranker feedback as in RaFe, translates the lay query into expert terms. We then run dense retrieval and sort the results by year.

Three ablations isolate the key ingredient. The first replaces the asymmetric vocabulary-gap cost with symmetric embedding distance, which keeps "stay topically close" but removes "don't introduce unexplained terms." The second sets $\alpha=0$, leaving only precedence. The third replaces the global solver with greedy nearest-neighbor chaining over identical costs.

All systems share the index, $L$, and a matched tuning budget of 50 configurations on a held-out SurveyBank split. Beating a baseline means a statistically significant gain in both gold-set recall and order agreement on ACL-rlg novice queries, with the gain growing as the vocabulary gap widens.

Why not just…?

Why not just rewrite the query into expert vocabulary and retrieve? This is a real threat to our recall claim. If the rewriter knows the jargon, one jump may be enough. It fails where the rewriter does not know the target vocabulary, and ACL-rlg's finding that LLM performance drops on 2023+ tutorials suggests exactly that failure. Rewriting also produces no order. The 2023+ split and the lay-paraphrase gap curve on SurveyBank settle this question. If rewriting matches us there, the stepping-stone retrieval claim fails and only the ordering claim survives.

Why not just ask GPT-4o for an ordered list? ACL-rlg documents hallucinated references and performance loss on newer topics. Our system can only return papers that exist in the index. The question is whether that grounding costs recall on older topics, and we report the split honestly.

Why not just sort by year, or topologically sort predicted prerequisites? Chronology is a strong proxy for reading order in some fields, so we include it as its own baseline on the same retrieved set. Topological sort inherits the cycle, thresholding, and input-order failures shown by BerTSP. The symmetric-cost and greedy ablations show whether the global asymmetric solve adds anything beyond these orderings.

Why not just use RePaGer's Steiner tree with better seeds? If RePaGer seeded with our pool matches our paths, the contribution reduces to the expansion step. We would then report it that way.

Proposed Experiments

The main benchmark is ACL-rlg with its novice-formulated queries. We measure Recall@20 and NDCG@20 against the expert list, plus Kendall's tau between our order and the tutorial's listed order for gold papers we retrieve. ACL-rlg is described as measuring ordered articles, but we will verify order annotations first. Where order is absent, we collect a small ordering annotation from two NLP experts.

The first decisive check is a stratified comparison on 2023+ tutorials. The prediction is that our margin over GPT-4o and LLM rewriting is largest there.

For scale, we use SurveyBank's 9,321 surveys with reference lists as gold sets. We generate lay paraphrases of each survey title at controlled levels of term overlap, which gives a dial on the vocabulary gap. The second decisive check is the slope: our advantage over rewriting and over the symmetric-cost ablation should grow as overlap falls. If it stays flat, vocabulary-gap edges are not doing the work.

The component check validates $P(i \to j)$ as a prerequisite predictor on LectureBank, where an SVM reached F1 up to 0.761. We also test on the University Course dataset, where AKD reports F1 0.842 against 0.767 for BERT. We do not need to beat AKD, only to show that the self-supervised citation signal is competitive.

Finally, we report the solver's optimality gap against exact branch-and-cut at pool sizes of 50 or fewer, and the wall-clock time against RePaGer's roughly one-minute average. A small reading study with novice participants, who answer comprehension questions after following paths from each system, tests whether a lower vocabulary gap actually means easier reading.