Search engines usually score a query against a document by how similar the two texts are, and similarity is symmetric: A resembles B exactly as much as B resembles A. Retrieval is not symmetric. A good answer to "does Zoloft cause weight gain?" is a dense abstract about sertraline and body mass. That abstract should contain what the question asks for, while carrying far more detail than the question. The abstract uses words the question never uses. A general review of antidepressants is similar to the question but does not answer it. We propose to teach a biomedical retriever this one-way relation during domain pre-training. Queries and documents become regions rather than points. The score asks whether the document's region sits inside the query's region, and the training pairs come for free from the structure of the biomedical literature itself.
In concrete terms, we continually pre-train a biomedical bi-encoder (a retriever that embeds queries and documents separately, so millions of documents can be searched quickly) to output a Gaussian distribution per text. We train it on containment pairs mined from PubMed and the Medical Subject Headings (MeSH) vocabulary: an abstract contains its own sentences; an abstract falls under its indexed MeSH headings, including their lay entry terms such as "heart attack" for myocardial infarction; a heading falls under its parent. The key ingredient is using reversed pairs as hard negatives. The model is penalized for scoring "general contains specific" as highly as "specific inside general." The bet is that this gives a first-stage retriever that bridges lay wording to technical wording and prefers evidence that is more specific than the query over text that is merely on-topic or more general. It should also keep doing so for concepts outside the ontology it was trained from.
Core Claim
Our central claim is that the direction of the training signal is what current domain retrieval pre-training lacks. Adding it is cheap when the supervision comes from the corpus's own containment structure instead of from generated queries. Current domain-matched retrieval pre-training teaches models what goes with what. Domain-matched Pre-training Tasks for Dense Retrieval pre-trains on 65M synthetic question–answer pairs. Improving Biomedical Information Retrieval with Neural Retrievers pre-trains Poly-DPR on PubMed with expanded-title and reduced-sentence mapping objectives. Condenser primes the pooled summary token for retrieval. None of these ever tells the model which way the relation runs. All three score with inner products between point vectors.
Directional work shows the missing signal can be learned. Sentence Representations via Gaussian Embedding (GaussCSE) outputs a Gaussian per sentence and scores pairs with asymmetric Kullback–Leibler (KL) divergence. With reversed NLI pairs as hard negatives it predicts entailment direction with over 97% accuracy on SNLI and MNLI, though only in general-domain NLI and only with large labeled corpora. Learning Probabilistic Sentence Representations from Paraphrases (WLO) shows, without specificity labels, that distribution entropy tracks generality. Sentences with precise technical nouns get low entropy, and premises have lower entropy than hypotheses in 75.8% of SNLI pairs. HYDEN nests clinical reports inside chest X-ray distributions. That work is image-to-text, however, and its hyperbolic machinery is heavy.
We take GaussCSE's scoring and reversed-negative trick and WLO's entropy-as-specificity finding, and move them into domain retrieval pre-training. We replace labeled NLI with containment pairs mined from the corpus. We evaluate on the lay-to-technical gap rather than on NLI.
The file surfaces a real tension, and we resolve it explicitly. GaussCSE gives the entailing sentence the larger variance: in the informational view, a region is the set of facts a text carries. WLO finds specific sentences have lower entropy: in the model-theoretic view, a region is the set of situations consistent with a text. For retrieval we adopt the WLO convention. A lay query is broad and a technical abstract is a narrow region inside it. We score
$$s(q, d) = -\mathrm{KL}\big(\mathcal{N}(\mu_d, \Sigma_d)\,\|\,\mathcal{N}(\mu_q, \Sigma_q)\big),$$
where $\mu$ and $\Sigma$ (diagonal) are the encoder's predicted mean and covariance for query $q$ and document $d$. This score is high when the document's probability mass falls inside the query's region. It is low when the document is broader than the query, because a wide $\Sigma_d$ spills outside a narrow $\Sigma_q$. We also run the opposite convention as an ablation, since which one works is an empirical question the prior work leaves open.
Research Program
We start from an existing biomedical encoder and continue pre-training on PubMed abstracts. We keep the domain in isolation rather than mixing in web text, following the "drowning" argument of Durrani et al.. We are also following Männistö's finding that domain pre-training predicts downstream performance better than adaptation choices. Two small heads on the pooled representation output $\mu$ and $\log \Sigma$.
The pre-training pairs are all "specific inside general" and need no annotation and no text generation. There are three kinds. The first maps an abstract to one of its own sentences. Here the sentence is the broader region, since it carries a subset of the abstract's information. This is the directional counterpart of Poly-DPR's reduced-sentence mapping. The second maps an abstract to its MeSH headings, rendered as the preferred term or a randomly sampled entry term. Entry terms include consumer phrasings, so this is where the vocabulary gap is bridged. The third maps a MeSH heading to its ancestor headings.
For each positive pair $(g, x)$, with $g$ the general text and $x$ the specific text, the loss is contrastive:
$$\mathcal{L} = -\log \frac{\exp s(g, x)}{\exp s(g, x) + \exp s(x, g) + \sum_{x'} \exp s(g, x')},$$
where $s(x, g)$ is the same pair with roles reversed (the reversed-direction hard negative) and $x'$ ranges over in-batch and mined topical negatives. The reversed term is the one ingredient no existing retrieval pre-training uses. Our ablations isolate it.
Using MeSH only as a training-time source of direction is a deliberate response to the main limitation of Zero-Shot Medical Information Retrieval via Knowledge Graph Embedding (MedFusionRank). That system links queries to MeSH at test time, embeds the graph with symmetric Node2Vec cosine, and falls back to a character-level LSTM when a term is missing from the ontology. We want the encoder to absorb the hierarchy and generalize past it. This is analogous to how Scoring Lexical Entailment with a Supervised Directional Similarity Network (SDSN) learns directional word-level entailment that transfers to a disjoint test vocabulary (0.544 Spearman on the HyperLex lexical split versus 0.453 before). SDSN works on single words over static embeddings. We work on passages inside a pre-trained transformer. To test generalization directly, we hold out entire MeSH subtrees during pre-training.
One known risk shapes the design. MedNLI Is Not Immune shows that clinical NLI models learn a "hypernym means entailment" shortcut and fall from about 51% to 40% accuracy on adversarially filtered data. Training on MeSH hierarchies could teach exactly that shortcut. We therefore report accuracy separately on negated queries, where the direction of containment flips (from "no antidepressants" one may infer "no sertraline," not the reverse). We treat collapse there as a finding, not noise.
Baselines
The naive reference points are BM25 and the unmodified biomedical encoder used as a cosine bi-encoder with no retrieval pre-training. BEIR shows BM25 is a strong zero-shot baseline that generic dense retrievers often fail to beat out of domain. On BioASQ, Improving Biomedical Information Retrieval with Neural Retrievers reports BM25 at MAP 65.10, against 53.31 for standard DPR. We run BM25 with default parameters on identical corpora.
The strongest existing retrieval systems in the file are the following. Poly-DPR reports MAP 66.66 on BioASQ. MedFusionRank reports MRR 0.552 and nDCG@10 0.357 on NFCorpus, and Recall@1k 0.990 on SCIFACT; the last is near ceiling, so we will rely on nDCG@10 there, for which there is no reported result on Lacuna for SCIFACT from these systems. MedFusionRank is the closest competitor on the lay-to-technical question because it attacks the same gap with the same ontology. We compare against its reported numbers and, where the code allows, against a re-run on our splits. We count ourselves as beating it only if our first-stage retriever, used alone, exceeds nDCG@10 0.357 on NFCorpus. Its BM25-fused variant is compared against our model fused with BM25 in the same way.
The most important baselines are controlled, not published. Every one of them uses the same backbone, the same PubMed corpus, the same number of training pairs and steps, and the same hyperparameter search budget. The first is a symmetric point-vector bi-encoder trained on our pairs with cosine scoring. The second is an asymmetric point-vector model with separate query and document towers and dot-product scoring, trained with reversed-direction negatives. The third is our density model without reversed negatives. The fourth is our density model with the GaussCSE variance convention instead of WLO's. The fifth is a data-source ablation: the same density model trained on an equal number of LLM-generated lay queries per abstract instead of MeSH pairs, in the style of docT5query document expansion from BEIR and hypothetical-document methods like AutoMIR.
The second of these is the decisive competitor for the geometric part of the claim. A two-tower dot product can in principle express direction. If it matches the density model once it receives the same reversed negatives, then the Gaussians are unnecessary. The honest conclusion would then be that the negatives carry the effect.
Why not just…?
Why not just use BM25, or a BM25 hybrid? BM25 cannot succeed when relevant abstracts share no content words with the question. Yang et al. find this situation common. It is endemic to consumer health queries like those in NFCorpus. BM25 remains a real threat on expert-written benchmarks like BioASQ, where it trails Poly-DPR by only 1.56 MAP. So we report results stratified by query–document term overlap. The claim stands only if gains concentrate in the zero-overlap stratum and the hybrid still improves on BM25 in the high-overlap stratum.
Why not just do domain-matched retrieval pre-training with generated queries and point vectors, as PAQ and Poly-DPR do? This is the most serious alternative, and on aggregate metrics it may well match us. Its known weakness is drift: RaFe and AutoMIR both note that generated rewrites and hypothetical documents introduce wrong technical assertions. Nothing in its objective separates "document answers the query" from "document is a broader treatment of the query's topic." Our first and fifth controlled baselines settle this directly. If the generated-query point model ties us on NFCorpus and BioASQ and on the direction diagnostic below, the proposal has failed, and we will report it that way.
Why not just rerank with a cross-encoder, which sees both texts jointly and can model direction? BEIR shows cross-encoder reranking is the strongest zero-shot configuration, but it cannot do first-stage search over tens of millions of abstracts. Poly-DPR's full-PubMed setting has 19 million documents. Our model is a candidate generator, so the relevant question is whether better first-stage recall helps the reranker. We test this by reranking the top 100 candidates from each first-stage model with the same cross-encoder and measuring the change in downstream nDCG@10.
Why not just link queries to MeSH at test time, as MedFusionRank does? Test-time linking is bounded by ontology coverage and uses symmetric graph similarity. The held-out-subtree experiment measures exactly the regime where it breaks: queries whose concepts the ontology-derived signal never saw.
Proposed Experiments
The first check is a direction diagnostic built from MeSH. For a query phrased with a heading's lay entry term, the model should rank abstracts indexed under that heading's descendants above abstracts indexed only under its ancestors or siblings. We measure pairwise ordering accuracy, evaluated separately on seen and held-out MeSH subtrees and on a negated-query subset. The success criterion is that the density model with reversed negatives beats the asymmetric point-vector model with the same negatives on held-out subtrees. We will fix the size of this margin in advance, after a pilot estimate of seed-to-seed variance, rather than choosing it after seeing results.
The second check is zero-shot retrieval on NFCorpus (nDCG@10 and MRR, against MedFusionRank's 0.357 and 0.552), BioASQ (MAP, against Poly-DPR's 66.66 and BM25's 65.10), and SCIFACT (nDCG@10), each stratified by lexical overlap.
The third check is whether predicted entropy is monotone in MeSH tree depth, and whether lay queries receive higher entropy than their matched abstracts. This would replicate WLO's unsupervised specificity finding inside a specialized domain.
The decisive outcome is the conjunction of two results. The density model must be non-inferior to the generated-query point-vector baseline on the standard benchmarks. It must also be clearly better on the direction diagnostic for held-out subtrees. If only the reversed negatives matter, the program reduces to a simpler and still useful recipe: containment pairs with reversed negatives for any bi-encoder. The experiments are designed so that this outcome is identified cleanly rather than hidden.
References
- Domain-matched Pre-training Tasks for Dense Retrieval
- Improving Biomedical Information Retrieval with Neural Retrievers
- Condenser: a Pre-training Architecture for Dense Retrieval
- Sentence Representations via Gaussian Embedding
- Learning Probabilistic Sentence Representations from Paraphrases
- HYDEN: Hyperbolic Density Representations for Medical Images and Reports
- A Deep Fusion Model for Domain Adaptation in Phrase-based MT
- A Comparative Study of PEFT Methods for Python Code Generation
- Zero-Shot Medical Information Retrieval via Knowledge Graph Embedding
- Scoring Lexical Entailment with a Supervised Directional Similarity Network
- MedNLI Is Not Immune: Natural Language Inference Artifacts in the Clinical Domain
- BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models
- AutoMIR: Effective Zero-Shot Medical Information Retrieval without Relevance Labels
- WikiQA: A Challenge Dataset for Open-Domain Question Answering
- RaFe: Ranking Feedback Improves Query Rewriting for RAG