Literature-search assistants built on retrieval-augmented generation (RAG) work in two steps. A search model pulls a few passages from a large paper collection, and a language model writes an answer that cites them. These assistants are becoming the main way researchers cope with more than five million papers a year. Their answers are only as representative as what the search step returns. Language models asked to recommend references already favor famous papers far more than human authors do: Large Language Models Reflect Human Citation Patterns with a Heightened Citation Bias reports a median citation-count skew of +1,326 relative to the papers humans actually cited. A medical study, Do AI chatbots find what experts would?, finds that chatbot search misses studies that systematic-review experts select, and it traces the gap to a preference for popular, English-language work. This is sampling bias in the statistician's sense. A paper's chance of being retrieved depends on its fame and not only on its relevance, so any review the system writes rests on a skewed sample of the evidence.
We propose to correct this inside the search model rather than after it. We use back-translation, a trick from machine translation in which a reverse model generates the missing input for real outputs. Here the reverse model turns real papers into the search queries those papers would answer. The new ingredient is deciding which papers to back-translate. We sample them across citation tiers instead of in proportion to how often they appear in existing training data, and we pair each synthetic query with hard negatives: famous papers that sit nearby but do not answer the query. The retriever trained this way should keep its accuracy on queries whose answer is a famous paper while recovering niche papers that humans would have cited. The research question is whether this beats the obvious fix of down-weighting citation counts at ranking time.
Core Claim
Popularity bias in dense scientific retrieval comes partly from the training signal. Query–paper pairs used to train retrievers are drawn mostly from well-cited, well-described work, and the base model learns that "central" papers are safe answers. If that is right, retraining on back-translated queries whose source papers are stratified by citation tier, with popular hard negatives, should shrink the citation skew of retrieved references toward the human distribution. It should do so without lowering recall of the papers humans actually cite, and it should beat both untargeted back-translation and a tuned popularity penalty on the trade-off between skew and recall. The target is calibration to human relevance judgments, not anti-popularity. Human citation is itself subject to the Matthew effect, so a system that never surfaces famous papers is also wrong. The Algaba et al. finding matters because the models are worse than the humans.
Research Program
The method has three parts. Each borrows from a specific line of work and changes it.
The first part is a back-translation model for scientific queries. Back-translation took off in translation with Edinburgh Neural Machine Translation Systems for WMT 16, where synthetic source sentences from monolingual news gave 4.3 to 11.2 BLEU gains. Self-Alignment with Instruction Backtranslation carried it to instruction tuning by predicting the instruction behind a web document. That paper also supplies a constraint we take seriously: uncurated synthetic pairs hurt, and gains appeared only when the top 10–20% were kept after self-scoring. RESTful-Llama showed the same pattern for grounding in technical domains by back-translating executable API code into user intents. We reuse this recipe of a reverse model plus self-curation. The reverse model reads a paper's abstract and a sampled full-text passage from the OpenScholar datastore of 45 million open-access papers (OpenScholar) and writes a researcher-style question that the passage answers. Humpback trains a generator. We use the synthetic pairs to train a retriever, which none of the cited back-translation work does.
The second part is choosing which papers to back-translate, and it is the core of the proposal. Translation research found that random choice of monolingual text over-represents easy, frequent material. Back-Translation Sampling by Targeting Difficult Words and A Semantic Uncertainty Sampling Strategy for Back-Translation fix this by targeting rare words or high-uncertainty sentences. We transfer that idea from words to papers. We bin the datastore by log citation count within field and publication year. We then draw source papers so that each bin gets a fixed share of synthetic queries, and inside each bin we oversample papers the current retriever fails to recover from their own back-translated query. That failure is the retrieval analogue of "difficult words." Concretely, a paper $d$ in citation bin $b$ is drawn with probability proportional to $w_b \cdot (1 - r(d))$. Here $w_b$ is the target share for bin $b$, and $r(d)$ is the base retriever's top-20 recall of $d$ for its own query.
The third part is curation and contrast. Each synthetic query passes two filters, both scored by a language-model judge in the Humpback style. The source paper must actually answer the query. The query must be specific enough to rule out the field's generic answer, which guards against the known failure where reverse models write vague prompts. For each kept query we retrieve the base model's top famous papers, use the judge to discard any that also answer the query (so we do not create false negatives), and keep the rest as hard negatives. Following Tagged Back-Translation, synthetic queries carry a marker token during training so the retriever can tell them apart from real queries. The generator, OpenScholar-8B, stays frozen throughout, so any change in end answers comes from retrieval alone.
Baselines
The naive reference point is the unmodified OpenScholar retriever and reranker (OpenScholar) over the same datastore with the same frozen generator. The OpenScholar authors themselves note that it does not reliably retrieve the most representative papers. A closed-book reference, in which the model recommends citations without retrieval, reproduces the Algaba et al. setting, where median skew was +1,326. That number tells us whether adding retrieval already moves skew before any training.
The strongest existing competitor for end-to-end scientific answering is OpenScholar with its iterative self-feedback loop. It is the strongest because its loop already issues extra retrievals to fill evidence gaps, and that could plausibly surface niche papers without any retraining. The file gives no reported ScholarQABench correctness or citation-accuracy number for it (no reported result found on Lacuna for OpenScholar's absolute ScholarQABench scores), so we will rerun it ourselves. The other reference system is PaperQA (PaperQA), which reports 69.5% accuracy on LitQA against 66.8% for human experts. On LitQA, our retriever plugged into PaperQA's agent must stay within noise of 69.5% to count as not harming answers.
The most dangerous baseline is a popularity penalty applied at reranking time: subtract $\lambda \log(1+c_d)$ from each paper's score, where $c_d$ is its citation count. We tune $\lambda$ on the same development split with the same number of trials as our method's hyperparameters, then trace a full curve of skew against recall over $\lambda$. Beating it means our method sits strictly above that curve. That is, at matched skew it must recover more human-cited papers, with the largest gap expected when the correct answer is a famous paper, because the penalty hurts exactly those cases.
Two ablations isolate the key ingredient. In the first, the same number of back-translated queries with the same curation and hard negatives come from papers sampled in proportion to the natural corpus distribution. This is plain doc-to-query augmentation, and it tests whether stratified, difficulty-aware selection matters at all. In the second, the stratified queries are kept but the popular hard negatives are replaced by random negatives. All variants fine-tune the same bi-encoder for the same number of steps on the same GPU budget, and no evaluation-paper text is used to generate queries.
Why not just…?
Why not just penalize citation counts at ranking time? This is a real threat, and the experiments are built around it. A penalty is cheap and directly targets the skew metric. Its weakness is that it ignores content: it demotes the famous paper even when that paper is the right answer, and it cannot help a niche paper the retriever never placed in the candidate pool to begin with. A reranker cannot recover what the first stage missed. The skew–recall curve comparison settles it. If the penalty curve matches ours, we would report that the cheap fix suffices and the training-signal hypothesis is wrong. We also test the combination, since the two may stack.
Why not just back-translate the corpus uniformly? Uniform back-translation is a standard way to adapt retrievers. However, the translation literature on targeted sampling (difficult words; semantic uncertainty) shows that natural-frequency sampling under-trains on the hard tail. A natural-distribution draw also inherits the corpus's own skew. The first ablation tests this directly. If uniform augmentation closes most of the gap, the contribution shrinks to "adapt the retriever to science," and we will say so.
Why not rely on self-reflective or iterative retrieval? Self-RAG and OpenScholar's feedback loop decide when to retrieve more and whether evidence supports a claim. But each extra round queries the same biased retriever, so iteration plausibly re-draws from the same skewed distribution. Running OpenScholar with feedback on and off measures how much skew iteration removes by itself.
Why not just prompt the generator to prefer less-cited work? The attribution-bias studies (Evaluation of Attribution Bias in RAG; generator-aware follow-up) show that generators pick among the passages they receive with position and frequency biases. They can only choose from what retrieval hands them. We include a prompted variant as a cheap check, but we expect it to change which retrieved paper gets cited, not which papers get retrieved.
Proposed Experiments
The primary test is a citation-recovery task built on the Algaba et al. protocol. We take machine-learning papers posted after the retriever's training cutoff, mask each citation in context, and ask the system to retrieve the cited work from the datastore. Two metrics are reported side by side. The first is recall@20 of the human-cited paper, broken down by the citation tier of that paper. The second is skew, defined as
$$\Delta = \text{median}_{d \in R}\log(1+c_d) - \text{median}_{d \in H}\log(1+c_d),$$
where $R$ is the set of retrieved top-1 papers, $H$ is the set of human-cited papers, and $c_d$ is citation count. The log keeps a few superstar papers from dominating. A system that matches human citation behavior has $\Delta$ near zero.
The decisive result is lower $|\Delta|$ than every baseline at equal or higher overall recall, with recall on top-tier targets no worse than the base retriever. To test that the method transfers rather than overfits to machine-learning citation style, we repeat the evaluation on biomedical claims from SciFact-Open, which reports a drop of more than 15 F1 when verification moves to an open corpus. There we check whether fewer of the remaining errors involve fame-driven distractors. We also rebuild an included-studies test in the style of the medical chatbot study, using expert-selected trials as gold.
End-answer quality is measured on ScholarQABench for correctness and citation accuracy, and on LitQA through PaperQA. The method fails on the quality side if either drops beyond run-to-run variance. A final check asks whether the effect is causal. We vary the share of back-translated queries assigned to the lowest citation tier and test whether skew moves monotonically with that share. If skew stays flat while the share changes, the training-signal explanation is wrong.
References
- Large Language Models Reflect Human Citation Patterns with a Heightened Citation Bias
- Do AI chatbots find what experts would? Effects of model, user role, and sample size on study retrieval for medical questions
- Edinburgh Neural Machine Translation Systems for WMT 16.
- Self-Alignment with Instruction Backtranslation
- RESTful-Llama: Connecting User Queries to RESTful APIs
- OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs
- Back-Translation Sampling by Targeting Difficult Words in Neural Machine Translation
- A Semantic Uncertainty Sampling Strategy for Back-Translation in Low-Resources Neural Machine Translation
- Tagged Back-Translation
- PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
- Self-RAG: Self-reflective Retrieval Augmented Generation
- Evaluation of Attribution Bias in Retrieval-Augmented Large Language Models
- Evaluation of Attribution Bias in Generator-Aware Retrieval-Augmented Large Language Models
- SciFact-Open: Towards open-domain scientific claim verification