Search systems that represent text as vectors fail in specialized domains in a particular way. Most words in a clinical note, a 10-K filing or a product catalog mean what they mean on the web. A small set of terms does not: "RF" becomes renal failure, "yield" becomes a bond quantity, and a part number stops being noise. A general encoder gets the shared vocabulary right and the shifted terms wrong. An encoder adapted to the domain fixes the shifted terms but, trained on scarce text, can damage everything else.
Language modeling met this same tension decades ago and solved it by interpolation: mix a general model and a domain model, and let the mixing weight depend on how much evidence the domain model has. We propose bringing that idea into token-level retrieval. Each token's vector becomes a mix of a general and a domain-adapted vector. The mixing weight is set per word by a label-free measurement of how far that word's meaning moved between the two encoders, shrunk toward the general encoder when the word is rare in the domain corpus.
Core Claim
In late-interaction retrieval, a query and a document each keep one vector per token, and relevance sums each query token's best match among the document's tokens. ColBERT works this way (ColBERT). Our claim is that domain adaptation for such retrievers should be applied unevenly across the vocabulary, and that the right amount per word can be estimated without relevance labels.
For a word type $w$, let $\mathbf{g}(w)$ be a token vector from a general retrieval encoder and $\mathbf{d}(w)$ the corresponding vector from a domain-adapted copy, mapped into the general space. The final vector is
$$\mathbf{e}(w) = \lambda(w)\,\mathbf{d}(w) + (1-\lambda(w))\,\mathbf{g}(w), \qquad \lambda(w) = s(w)\cdot\frac{n(w)}{n(w)+\tau}.$$
Here $s(w)\in[0,1]$ is a measured semantic-shift score, $n(w)$ is the word's count in the unlabeled target corpus, and $\tau$ is one smoothing constant. The count factor is borrowed directly from history-dependent and Bayesian interpolation in n-gram modeling. Connecting and Comparing Language Model Interpolation Techniques shows that weights depending on counts, rather than one global weight, consistently beat static linear interpolation. Our formula transplants that count dependence from next-word probabilities to retrieval vectors. Documents are encoded once with the blended vectors, so the result is a single index with the same storage cost as one ColBERT index. Fusing the scores of two retrievers would need two indexes.
The novelty sits in three choices together: the interpolation happens at the token vector level, not at the score or model level; the weight comes from a shift measurement on unlabeled domain text, not from relevance supervision; and rare words back off to the general encoder.
Relation to Prior Work
Three lines of work motivate the gap.
The first is language model interpolation. Experiments in Domain Adaptation for Statistical Machine Translation mixes Europarl and News Commentary language models and finds a U-shaped perplexity curve as the global weight varies. Data selection and smoothing in an open-source system for the 2008 NIST machine translation evaluation shows that interpolating per-source models beats pooling the data, lowering perplexity from 189.3 to 161.7. Alignment Model Adaptation for Domain-Specific Word Alignment goes further and weights interpolation by relative word frequency, beating data concatenation. All three work with discrete count models. We reuse their logic of per-word, evidence-dependent weights in a continuous embedding space.
The second is evidence that meaning shifts locally. Exploiting Task-Oriented Resources to Learn Word Embeddings for Clinical Abbreviation Expansion reports that PubMed embeddings reach only 28.06% accuracy on ICU abbreviation expansion, against 82.27% for enriched domain embeddings. Rare and out-of-vocabulary terms cause 81% of that paper's remaining errors. That finding is exactly why we include the count-based backoff. Word Embeddings-Based Uncertainty Detection in Financial Disclosures shows that SEC-trained embeddings double volatility predictiveness relative to Google News embeddings. Both papers use static, pipeline-style embeddings. Neither offers a way to keep general knowledge for words that did not shift.
The third line adapts dense retrievers, and it constrains our design. Disentangled Modeling of Domain and Relevance for Adaptable Dense Retrieval (DDR) trains relevance once and swaps in a domain backbone trained with masked language modeling. That recipe supplies how we build the domain encoder, but DDR switches backbones wholesale rather than blending them. Search-Adaptor learns one global residual correction on frozen embeddings using labeled ranking data; we instead vary the correction by word and need no labels. Learning Word Embeddings with Domain Awareness softly weights target-domain co-occurrence during word2vec training. It is the closest analogue of a per-word domain weight, but it is static and is never measured against retrieval. TRIAL adds token-importance and token-relation gates to late interaction and reports weaker results on LoTTE-Forum. That paper also warns that subword fragmentation breaks domain terms apart, which shapes how we compute shift scores.
Research Program
We first build two aligned encoders. The general encoder is a ColBERT-style model trained on MS MARCO. Following DDR, the domain encoder is the same backbone after continued masked language modeling on the unlabeled target corpus, with the relevance-trained layers reused unchanged. The two spaces are not guaranteed to match, so we fit an orthogonal map from the domain space to the general space. The map is estimated by least squares, using the contextual vectors of words whose neighborhoods overlap most across the two encoders. In practice these are mostly function words and generic vocabulary.
Second, we measure shift without labels. For each whole word in the target corpus, we average its contextual vectors under each encoder. We then compute $s(w)$ as one minus the overlap between its $k$ nearest neighboring words in the general space and in the domain space. Neighbor overlap needs no alignment, so the shift score does not depend on the map fitted in the first step. To avoid the fragmentation problem TRIAL describes, we compute scores on whole words and assign the same weight to every subword piece of a word. A variant also computes shift per occurrence, so that a word like "RF" can be weighted differently in different contexts.
Third, we validate the shift score itself, before any retrieval experiment. It should rank known shifted terms highly: the clinical abbreviations and financial uncertainty words discussed in the two papers above, where such lists can be reconstructed. It should rank function words near zero. If it does not, the method has no basis and we stop.
Fourth, we index and retrieve with the blended vectors, tuning only $\tau$ and $k$ on a small development split per domain.
Baselines
The naive reference points are the general encoder alone ($\lambda=0$) and the domain encoder alone ($\lambda=1$). Any proposal that fails to beat the better of the two on each domain is worthless.
The strongest existing competitor for unlabeled adaptation is DDR, which is effectively our domain encoder used as a full replacement. Its authors report a 20% to 50% relative improvement over standard dense retrievers on LoTTE and wins over BM25 on Lotte-Tech and Lotte-Sci. We will run it from the same checkpoint, with the same masked-language-modeling steps on the same corpus. Beating it means a consistent nDCG@10 gain across LoTTE domains with no losses on the general-domain MS MARCO check.
For absolute context, TRIAL reports average nDCG@10 of 72.15 on LoTTE Search and 51.09 on BEIR. We will reproduce TRIAL under our training budget rather than compare against its reported numbers directly.
The strongest score-level competitor fuses the two retrievers' final scores with a tuned global weight. We also run a hybrid with BM25. On TREC Product Search, the hybrid f_splade_bm25 reaches NDCG@10 of 0.7505 against 0.6540 for BM25 (TREC 2023 Product Search), so any dense method there must be reported both alone and fused with BM25.
Search-Adaptor serves as a supervised reference. Its authors report gains of over 5% average nDCG@10 on BEIR. We train it on the same small labeled development split we use for tuning, so it is clear what labels buy.
Three ablations isolate the key ingredient. A global-$\lambda$ version uses one tuned weight for all words. A shuffled version randomly permutes the $\lambda(w)$ values across the vocabulary, preserving their distribution but destroying which word gets which weight. A no-backoff version sets the count factor to one. Finally, a per-query-token score-mixing variant blends the two encoders' best-match scores with $\lambda$ instead of blending vectors. It tests whether vector-level blending matters beyond score-level blending, at the cost of a second index.
Why not just…?
Why not just continue pre-training the encoder on domain text and use it alone? This is DDR, and it is a serious threat. The case against it is indirect: domain-only training overfits when text is scarce (Learning Word Embeddings with Domain Awareness), and our design reasons that unshifted vocabulary should not pay that cost. The $\lambda=1$ baseline and DDR comparisons settle this directly. We will also vary the size of the domain corpus. We expect our advantage to grow as domain text shrinks, and if it does not, the backoff term is unjustified.
Why not just fuse the scores of the general and domain retrievers? To Interpolate or not to Interpolate shows that score interpolation helps, reaching 0.4858 MAP versus 0.4590 for dense-only retrieval on DL19 when interpolation enters at both stages of its feedback pipeline. It needs two indexes and applies one weight to every query. A tuned global fusion may still match us. If it does at equal cost, our claim reduces to a storage saving, and we will report it that way.
Why not just learn a per-token gate from relevance labels? A learned gate may well outperform a measured shift score whenever labels exist. Our target setting is the case where they mostly do not. We will train such a gate on the same small development split and report how many labeled queries it needs to overtake the label-free gate. The shuffled-$\lambda$ control separately tests whether the shift score carries any signal at all.
Proposed Experiments
Primary evaluation uses LoTTE Search and Forum (Tech, Sci and the other domains), measured by nDCG@10 and Success@5. We add BEIR's biomedical and scientific subsets and the TREC 2023 Product Search track (NDCG@10). We include UDA's FinHybrid and PaperText as retrieval-only tasks with gold evidence chunks, scored by recall@k. ClimRetrieve is a stress test where text-embedding-3-large peaks at F1 near 0.179 (ClimRetrieve). The same blended model is also evaluated on MS MARCO to check that general retrieval is not degraded.
The decisive checks are three. First, shift-gated blending must beat both the shuffled-$\lambda$ and global-$\lambda$ ablations on a majority of domains with paired significance. If shuffling costs nothing, the per-word story is false. Second, it must match or beat DDR in-domain while staying within noise of the general encoder on MS MARCO. Third, its gains should concentrate on queries containing high-shift words: we split queries by their maximum $s(w)$ and expect the improvement to rise with shift and vanish for low-shift queries. Without that pattern, any aggregate gain would be coming from somewhere other than the mechanism we propose.
References
- ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
- Connecting and Comparing Language Model Interpolation Techniques
- Experiments in Domain Adaptation for Statistical Machine Translation
- Data selection and smoothing in an open-source system for the 2008 NIST machine translation evaluation
- Alignment Model Adaptation for Domain-Specific Word Alignment
- Exploiting Task-Oriented Resources to Learn Word Embeddings for Clinical Abbreviation Expansion
- Word Embeddings-Based Uncertainty Detection in Financial Disclosures
- Disentangled Modeling of Domain and Relevance for Adaptable Dense Retrieval
- Search-Adaptor: Embedding Customization for Information Retrieval
- Learning Word Embeddings with Domain Awareness
- TRIAL: Token Relations and Importance Aware Late-interaction for Accurate Text Retrieval
- OVERVIEW OF THE TREC 2023 PRODUCT PRODUCT SEARCH TRACK
- To Interpolate or not to Interpolate: PRF, Dense and Sparse Retrievers
- ClimRetrieve: A Benchmarking Dataset for Information Retrieval from Corporate Climate Disclosures