Dense retrievers turn every document into a vector of several hundred floating-point numbers. That works well until the collection is large. Storing PubMed-scale literature as 768-dimensional float vectors takes tens of gigabytes of memory, and standard database compression cannot shrink such vectors because they look like random noise to general-purpose compressors (Data formats in analytical DBMSs). The usual fix is to quantize the vectors, replacing each one with a short code. Separately, retrievers built for technical fields like medicine only become competitive when they are pre-trained on in-domain text and training signals. Today these two steps run in sequence. A model is first specialized to the domain, and a compression step is then fitted onto whatever vector geometry the specialization happened to produce.
We propose to reverse the order at the point where it matters most: very small memory budgets, far below the 30x ratio at which existing joint quantizers already match uncompressed quality. The idea is to put a nested code bottleneck inside the domain pre-training objective itself. Every citation-linked pair of PubMed papers is then scored through its 8-, 16-, 32- and 64-byte codes as well as its full vector. The encoder therefore learns to place domain distinctions, such as rare genes, drug names and disease subtypes, in the directions that survive coarse quantization. Our hypothesis is specific. At budgets of 16 to 32 bytes per passage, compression-aware domain pre-training will beat domain pre-training followed by the strongest joint quantization fine-tuning, when both use the same compute. At 96 bytes and above, it will not beat it. The second half of that hypothesis is a planned null result, not a hedge.
What exists and where the gap is
The closest work is Domain Adaptation for Memory-Efficient Dense Retrieval. It showed that supervised binary hashing and product quantization trained on MS MARCO lose up to 14% nDCG@10 when used zero-shot on BEIR. It then recovered that loss by fine-tuning the compressed TAS-B retriever on synthetic domain queries scored by a cross-encoder, a method called GPL. The adapted JPQ+GPL reaches 0.435 average nDCG@10 on BEIR against 0.415 for uncompressed TAS-B.
That work motivates ours, but its adaptation is a late fine-tuning stage on a general backbone. Its compression budget is also fixed at roughly 32x. We change both the stage where compression enters (pre-training) and the budget regime (100x or more).
Several papers supply the methods we reuse.
- JPQ showed that product quantization trained against a ranking loss reaches 0.341 MRR@10 on MS MARCO, compared with 0.123 for post-hoc PQ. We reuse its asymmetric setup, in which queries stay full-precision and centroids are trained by gradient.
- BPR learns binary codes end-to-end with a tanh continuation. It shrinks a Wikipedia index from 64.6GB to 2.0GB at 41.6 EM on NQ versus 41.5 for DPR.
- Matryoshka Quantization nests low-bit integers inside the most significant bits of higher-bit ones and trains all widths jointly. It gains up to 7% at int2 for LLM weights. We borrow the nesting idea but apply it to retrieval embeddings rather than weights.
The domain side supplies the training signal. BiCA trains a 110M encoder on 20,000 PubMed articles with hard negatives drawn from 2-hop citation walks. It reaches 0.518 average nDCG@10 on BEIR, above GTR-xxl's 0.486, but it only evaluates full-precision vectors. Domain-matched Pre-training Tasks for Dense Retrieval shows that pre-training on retrieval-shaped pairs matters more than model size.
Two papers constrain the design.
- Isotropic Iterative Quantization finds that in text embeddings the high-variance directions often encode frequency artifacts, while the weak directions carry meaning. Naive variance-driven quantization therefore throws away the wrong information. Technical vocabularies have heavy long tails, so this concern should be sharper in biomedicine. This is the mechanism we expect pre-training to fix and post-hoc rotation to fix only partly.
- BEIR and the ECNU-ICA TREC 2019 Precision Medicine system both find that BM25 often beats unadapted neural retrievers in technical domains. Any tiny-index method must therefore clear BM25, not just other dense models.
Core Claim
Post-hoc and fine-tuning-stage quantizers can only redistribute the information an encoder already put into its leading quantizable structure. At extreme budgets, the codebooks run out of capacity before they reach the low-variance directions that encode rare domain entities.
If the codes are present during domain pre-training, gradients from millions of citation pairs push those distinctions into the coarse codes. We therefore predict three things.
- The gap between compression-aware pre-training and fine-tuning-stage quantization should grow as the budget shrinks.
- The gap should be larger on rare-entity queries than on frequent-entity queries.
- The gap should mostly disappear when the same bottleneck pre-training is run on general web text, because the effect depends on domain structure and not on code nesting alone.
Research Program
Base model and corpus. We start from the same general 110M encoder that BiCA uses. We continue pre-training it contrastively on PubMed, using title–abstract pairs and citation pairs, with 2-hop citation hard negatives mined as in BiCA.
Code bottleneck. Document vectors pass through residual quantization with $S=8$ stages. Each stage chooses one of 256 centroids, so it costs one byte, and every prefix of stages forms a valid smaller code. Let $\hat d_s$ be the reconstruction of document $d$ from its first $s$ stages. The query $q$ stays full-precision, and $\mathcal{L}(q,\cdot)$ is an in-batch contrastive loss with citation hard negatives. The pre-training objective is
$$L = w_0\,\mathcal{L}(q, d) + \sum_{s \in \{1,2,4,8\}} w_s\,\mathcal{L}(q, \hat d_s),$$
where $w$ weights the budgets and $\mathcal{L}(q,d)$ uses the full vector.
To reach the target budgets of 8 to 64 bytes, each stage operates on 16 sub-spaces, following JPQ's product structure. This makes a one-stage code 16 bytes.
Centroids are updated by gradient. Code assignments are frozen for stretches of training and reassigned periodically, as JPQ does after warm-up. Gradients reach the encoder through straight-through estimation.
Shared fine-tuning. After pre-training, every system receives an identical GPL fine-tuning stage on each target corpus. This isolates the effect of the pre-training step.
Analysis. We will measure how much variance each code prefix captures. We will also measure how often rare and frequent UMLS-style entities in queries are preserved in the nearest neighbors retrieved with the coarse codes, compared with the full vectors. This tests the isotropy mechanism directly rather than only through end metrics.
Baselines
Naive references. The "do nothing" reference is BM25, which costs no vector memory and is often strong in these domains according to BEIR. The upper reference is uncompressed BiCA-style domain pre-training plus GPL. BiCA reports 0.518 average nDCG@10 on BEIR. No reported result was found on Lacuna for its per-dataset biomedical scores, so we will re-run it on our own splits.
Strongest existing method. This is domain pre-training followed by JPQ joint quantization and GPL fine-tuning at each byte budget. It combines the two best available ingredients in the conventional order. The closest published number is JPQ+GPL on a general TAS-B backbone at 0.435 average nDCG@10 on BEIR, from Domain Adaptation for Memory-Efficient Dense Retrieval, at roughly 32x compression. No reported result was found on Lacuna for any method at 16 or 32 bytes per passage. We also run BPR-style binary hashing at matched bytes.
Fairness controls.
- Every dense system gets the same PubMed pairs, the same total pre-training plus fine-tuning steps, and the same GPL query budget.
- Hyperparameters are tuned with equal search budgets on a held-out biomedical dev set.
- Codebooks for every system are fit on the same target corpora.
What counts as winning. Beating the strongest baseline means a mean nDCG@10 gain of at least 2 points at 16 and 32 bytes on the biomedical BEIR sets and TREC 2019 Precision Medicine. The gain must be significant under paired tests across queries. It must also come with no loss at 128 bytes.
Ablations.
- The same bottleneck pre-training on general web-style pairs, which isolates the domain.
- Domain pre-training with the bottleneck applied only in the final fine-tuning stage, which isolates the stage.
- IIQ rotation applied post hoc to the domain encoder, which tests whether geometric realignment alone captures the gain.
Why not just…?
Why not just domain-pretrain, then run JPQ with GPL? This is the main threat. At about 32x it probably does match full precision, which is why our claim is restricted to 100x and above. The byte-budget sweep settles the question directly. If the curves of the two systems coincide down to 16 bytes, the hypothesis fails and we report that compression can safely be deferred.
Why not just keep full vectors on disk and rerank a coarse candidate list, as BiDR does? BiDR reaches +17.45% Recall@10 over STAR+DiskANN on one billion ads with 32GB of RAM. Two-tier storage makes precision at the top of the ranking cheap, so the coarse code only needs recall. We therefore also report Recall@1000 of the RAM-resident code alone, with an identical disk-resident rerank for all systems. Our method must win on recall at a fixed RAM budget, not only on nDCG from the codes.
Why not just use BM25 or reduce dimensionality? BM25 needs no vector memory, and BEIR shows that it frequently beats zero-shot neural retrievers. PCA truncation or IIQ rotation to 32 bytes is equally simple. Both are included at matched memory. If BM25 matches our 16-byte codes on the biomedical sets, the tiny-index regime has no practical value and we will say so.
Proposed Experiments
Test data. The main tests use the biomedical and scientific datasets in BEIR and the MEDLINE abstract track of TREC 2019 Precision Medicine. For the latter, ECNU-ICA reports infNDCG 0.4672 and P@10 0.5675 for its re-ranked abstract run, and we will compare against those.
Sweep and metrics. We sweep budgets of 8, 16, 32, 64 and 128 bytes. We report nDCG@10, Recall@1000, index size and CPU query latency.
Decisive checks. There are three.
- The quality gap between our model and domain-pretrain-then-JPQ should widen monotonically as the budget shrinks.
- The gap should be larger on queries whose key entity is rare in PubMed.
- The general-corpus bottleneck ablation should recover less than half of the gain.
If none of the three holds, the argument for moving compression into pre-training fails.
Generality check. A secondary check applies the same recipe to general-domain BEIR sets to see whether the effect is specific to the domain.
References
- Data formats in analytical DBMSs: performance trade-offs and future directions
- Domain Adaptation for Memory-Efficient Dense Retrieval
- Jointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performance
- Efficient Passage Retrieval with Hashing for Open-domain Question Answering
- Matryoshka Quantization
- BiCA: Effective Biomedical Dense Retrieval with Citation-Aware Hard Negatives
- Domain-matched Pre-training Tasks for Dense Retrieval
- Embedding Compression with Isotropic Iterative Quantization
- BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models
- ECNU-ICA team at TREC 2019 Precision Medicine Track
- Progressively Optimized Bi-Granular Document Representation for Scalable Embedding Based Retrieval