Dense retrievers compress each document into one fixed-length vector, a list of a few hundred numbers compared with the query by a dot product. This is fast and easy to index. It also means every document must fit through the same narrow channel. Pre-training schemes that train this vector directly, such as Condenser and SimLM, work the same way: a small decoder must rebuild the document from the vector alone, and the vector learns whatever that decoder needs. Both were pre-trained only on Wikipedia and web text. Yet dense retrievers fail most badly in technical domains such as biomedicine and finance, where BM25 keyword search often beats them (BEIR).
Our idea rests on a simple observation. The vector only has to carry the information the decoder cannot already predict. Cramming 1568 Tokens into a Single Vector supports this: how much text a single vector can hold depends on the text's cross-entropy under the decoder, not on its token count. In biomedical text, much of each abstract is predictable domain boilerplate ("patients were randomized", "expression was upregulated"). The parts that tell one document from another are rare entities, numbers and specific claims. A reconstruction objective that weights every token equally makes the vector spend capacity on the shared boilerplate. We propose to weight reconstruction by each token's surprisal under a domain language model, so the vector's fixed budget goes to the document-specific residual. The claim is that this makes bottleneck pre-training on domain corpora produce better single-vector retrievers than domain pre-training with uniform weighting, and that the domain model's notion of surprise is what matters.
Core Claim
Let $x_1,\dots,x_T$ be a document's tokens and $h$ the encoder's single output vector. Let $s_t = -\log p_{\text{dom}}(x_t \mid x_{<t})$ be the surprisal of token $t$ under a frozen, domain-pretrained causal language model, meaning how unexpected that token is to a model fluent in the domain. We train the encoder and a shallow decoder $p_\phi$, which sees only $h$ and a heavily masked copy of the text $\tilde{x}$, with
$$ \mathcal{L} = -\sum_{t=1}^{T} w_t \log p_\phi(x_t \mid \tilde{x}, h), \qquad w_t = \frac{\min(s_t,\tau)}{\frac{1}{T}\sum_{u}\min(s_u,\tau)}, $$
where $\tau$ is a clipping threshold that stops tokenization artifacts from dominating the loss. The normalization keeps the average weight at one, so total gradient scale matches the uniform SimLM loss.
The rationale follows from the Cramming result. The bits $h$ must store are roughly the decoder's remaining cross-entropy. Tilting the loss toward tokens that a domain-fluent model finds surprising pushes those bits toward the content that separates documents.
Two predictions follow and are testable. First, weights from a domain model should beat weights from a general web model. A general model finds domain jargon surprising, so its weights still pay the vector to encode vocabulary that every document shares. Second, weights from a domain model should beat plain inverse document frequency weighting, which cannot tell a predictable rare term in context from a surprising one.
Relation to Prior Work
Condenser and SimLM supply the method. They show that token-level masked language modeling leaves the summary token unprepared, and that a skip-free decoder conditioned on it fixes this. We keep their architecture and change two things: the corpus becomes domain text, and the reconstruction weights become non-uniform.
Domain pre-training work motivates the regime. BioBERT and BioBART show that pre-training on PubMed gives representations general models lack. BioBART also found that sentence permutation hurts biomedical generation, a warning that scientific discourse order carries meaning. We therefore corrupt by masking only, never by reordering. Neither line of work trains a sequence-level vector.
Post-hoc adaptation constrains us, because it is the strongest competitor. GPL generates synthetic queries and pseudo-labels them with a cross-encoder, gaining up to 9.3 nDCG@10 on BEIR targets. The file reports that GPL does best when combined with an unsupervised denoising-autoencoder pre-training step on the target corpus. That means "bottleneck pre-training on domain text" is not by itself new. What is untested is whether the reconstruction target should be shaped by domain predictability.
Domain Adaptation for Memory-Efficient Dense Retrieval shows that binary hashing loses up to 14% nDCG@10 zero-shot on specialized domains, and that GPL recovers it. This gives us a stress test. If our vectors concentrate document-specific content, they should degrade less under compression. Rep2Text recovered only 13% of out-of-distribution clinical notes with high fidelity from single hidden states. This suggests that off-the-shelf summary vectors lose domain specifics, which our probes will measure directly.
Research Program
We build biomedical encoders at BERT-base scale, initialized from a biomedical masked-language-model checkpoint in the BioBERT style. Each is pre-trained on the same budget of PubMed-abstract tokens under five objectives: continued masked language modeling, uniform Condenser, uniform SimLM-style reconstruction, and our weighted reconstruction under three weightings (domain-LM surprisal, general-LM surprisal, inverse document frequency). The domain LM is a small causal model trained on the same abstracts, so it adds no outside data.
Every encoder then gets identical contrastive fine-tuning on MS-MARCO. We evaluate each with and without GPL adaptation on the target corpora.
Alongside retrieval, we run a diagnostic that ties the result to the bottleneck mechanism. We train a probe decoder for each frozen encoder and measure reconstruction accuracy from $h$ separately for high-surprisal tokens (entities, numbers, gene and drug names) and low-surprisal tokens. On a small sample of PubMed and PG-19 passages, we also run the Cramming per-sample optimization. This extends its capacity measurement to domain text and gives the "encoder gap" in a domain, which the file lists as unmeasured.
Baselines
The naive reference points are BM25 and the continued-masked-language-model encoder with the same fine-tuning. BM25 is included because BEIR finds it frequently beats zero-shot dense models out of domain. The continued-MLM encoder shows what domain pre-training without any bottleneck buys.
The strongest existing method is a strong dense retriever adapted with GPL, following GPL and its denoising pre-training combination. The file reports two relevant figures. First, GPL gains up to 9.3 nDCG@10 over non-adapted models. Second, in Domain Adaptation for Memory-Efficient Dense Retrieval, JPQ+GPL averages 0.435 nDCG@10 against 0.415 for uncompressed TAS-B. The file has no per-dataset numbers: no reported result found on Lacuna for TREC-COVID, NFCorpus, SciFact, BioASQ or FiQA individually. We will therefore re-run GPL on each target ourselves, using the same generated queries, the same cross-encoder teacher and the same number of steps for every encoder. The comparison that matters is our encoder plus GPL against uniform bottleneck plus GPL. Beating it means a mean nDCG@10 gain across the biomedical BEIR sets that exceeds the spread across three seeds.
Uniform domain SimLM and domain Condenser are the ablations that isolate our key ingredient. Everything is shared except the weights $w_t$. All pre-training variants get identical token counts, learning-rate sweeps of equal size, and the same fine-tuning recipe. Reference numbers for the uniform methods exist only in the general domain: SimLM reports MRR@10 of 41.1 on MS-MARCO and Condenser reports 0.366. We report MS-MARCO for all variants so we can check that the weighting does not trade away in-domain general performance.
Why not just…?
Why not just use BM25, or a BM25-plus-dense hybrid? Surprisal weighting partly rediscovers what BM25 already does: it rewards rare, distinguishing terms. That makes a hybrid a real threat. LexRAG found that dense models beat BM25 when users describe intent without formal terminology, and Contriever notes that BM25 fails when query and document wording differ. Still, a hybrid could absorb our gains. We will therefore report every encoder both alone and interpolated with BM25. If our advantage over uniform SimLM disappears once BM25 is added, the vector is only re-learning lexical matching, and we will say so.
Why not just weight by inverse document frequency? It is cheaper, and it needs no language model. It is built in as an ablation. If inverse document frequency matches domain-LM surprisal, the central claim about domain predictability fails and the method reduces to a simpler trick. We expect surprisal to win on sets like SciFact, where the same rare term can be expected in one sentence and informative in another. The probe decoder will show whether the two weightings put different content into $h$.
Why not just run GPL on a uniformly pre-trained domain encoder? This is the closest competitor, and GPL is strong. However, GPL's own finding that denoising pre-training adds to pseudo-labeling means the pre-training objective still matters after adaptation. Our design crosses every objective with GPL on and off. If the gap between weighted and uniform objectives closes to zero after GPL, then the weighting only substitutes for adaptation. That would still be useful when cross-encoder labeling is too costly, but it would be a weaker result.
Proposed Experiments and Decisive Checks
The primary evaluation is nDCG@10 on TREC-COVID, NFCorpus, SciFact and BioASQ from BEIR. To test transfer outside biomedicine, we replicate the pipeline on FiQA with a finance corpus and a finance language model. Robustness to compression is measured by applying JPQ and BPR hashing to every encoder, following Domain Adaptation for Memory-Efficient Dense Retrieval, and comparing each encoder's loss relative to its own uncompressed version. As a stretch, the legal setting uses LexRAG Recall@10, against the reported 33.33%, run with the LexiT toolkit.
Three results would decide the question. First, the domain-surprisal weighting must beat both general-LM surprisal and inverse document frequency. Without that, the domain-knowledge story is wrong. Second, the probe decoder must show that the weighted encoders reconstruct high-surprisal tokens better while reconstructing low-surprisal tokens worse. This is the capacity trade-off the mechanism predicts, and without it any retrieval gain lacks the proposed explanation. Third, the retrieval gains should concentrate on queries whose relevant documents differ from hard negatives mainly in rare entities. We identify these queries by the entity overlap between each relevant document and the top-ranked negatives.
References
- Condenser: a Pre-training Architecture for Dense Retrieval
- SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval
- BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models
- Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the Limits of Embedding Space Capacity
- BioBERT: a pre-trained biomedical language representation model for biomedical text mining
- BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model
- GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval
- Domain Adaptation for Memory-Efficient Dense Retrieval
- Rep2Text: Decoding Full Text from a Single LLM Token Representation
- LexRAG: Benchmarking Retrieval-Augmented Generation in Multi-Turn Legal Consultation Conversation
- Unsupervised Dense Information Retrieval with Contrastive Learning