Topic Gaussians That Stretch Across Registers: Demonstration Retrieval Under Vocabulary Mismatch

Large language models can learn a task from a handful of worked examples placed in the prompt. Which examples get picked matters a great deal: a small model given retrieved examples can beat a much larger one given random examples. Almost every retriever used for this picks examples whose embedding vectors sit close to the query. That works when the query and the example pool use the same language. It breaks when they do not, for instance when a layperson asks about "chest tightness after climbing stairs" and the pool of medical exam questions says "exertional angina." We propose to represent the example pool as a set of topics, each modeled as a Gaussian distribution (a cloud with a center and a shape) in embedding space. The shape of each cloud is fitted so that it stretches along the direction separating everyday phrasing from expert phrasing. A lay query can then fall inside the right topic even when it is far from every individual example. The retriever then spends its few prompt slots on topics where the model is weak, using typical, central examples from those topics rather than outliers.

In two sentences: retrieval under vocabulary mismatch fails because it compares the query to single example points. Comparing it to topic clouds, whose covariance is fitted on both the pool and unlabeled incoming queries, should route mismatched queries to the right topic cheaply, with no rewriting of the query and no per-example query generation. The test is whether this beats nearest-neighbor retrieval, a discrete-topic retriever, and LLM query rewriting on medical and legal few-shot tasks, with the advantage growing as word overlap between query and pool shrinks.

Core Claim

The closest existing system is TopicK. It mines flat, discrete topics from the demonstration pool and trains a small classifier to predict a query's topics. It measures the target model's zero-shot accuracy on each topic and picks demonstrations covering topics where the model is weak. It reports gains of up to 6.38% on MedMCQA and Law and runs 37 times faster than the uncertainty-based ConE. Its topic classifier, though, is trained only on pool text. It has no way to represent how far a query sits from a topic, or in which direction. So it cannot tell a query that is off-topic from one that is on-topic but worded differently.

The second close work is PQR. It fights vocabulary mismatch in passage retrieval by generating about 300 queries per document with an LLM and fitting a Gaussian mixture to their embeddings. It then scores only by dot product against the component means, throwing away the covariances it estimated.

Our claim sits between the two. For demonstration selection, the covariance of topic-level Gaussians, adapted to unlabeled incoming queries, carries the information needed to bridge the register gap. That information costs one mixture fit per topic, not hundreds of LLM generations per document. It differs from TopicK in using continuous densities with an explicit shape and in adapting to the query side. It differs from PQR in operating at the topic level for demonstration selection, in actually using the covariance at scoring time, and in using no generated queries at all.

Research Program

We first encode every pool input and every unlabeled test input with an off-the-shelf contextual sentence encoder. We then fit $K$ topic Gaussians on the pool alone by expectation-maximization, which alternately assigns points to topics and re-estimates each topic's center and shape. Full covariances are too costly in 768–1024 dimensions, a limitation noted in Sentence Representations via Gaussian Embedding, which falls back to diagonal covariances. We therefore use a low-rank-plus-diagonal covariance with rank 16–32. Inverting it and computing its determinant is cheap.

The key step is a transductive adaptation. The pool fit serves as a conjugate Normal–Inverse-Wishart prior, the same conjugate structure Gaussian LDA relies on. We then update each topic's covariance on the unlabeled test inputs through a maximum-a-posteriori step, meaning the pool fit anchors the estimate and the query data adjusts it. We hold the means close to the pool means. If lay queries sit systematically offset from expert text, this lets the offset show up as an elongated axis of the existing topic, rather than as a new topic. We expect the adaptation to work because of Ma et al.. That paper showed that Gaussian likelihoods over embeddings classify short texts correctly even when more than 43% of test words were never seen in training. Its embeddings were static; ours are contextual, and our Gaussians are adapted to the query side.

Next, we estimate the model's competence per topic. As in TopicK, we record the target model's zero-shot correctness on pool examples. Instead of hard topic assignments, we weight each example by its posterior probability of belonging to each topic, which gives a soft competence $a_k \in [0,1]$ for every topic $k$. A candidate demonstration $d$ with embedding $e_d$ is scored against a query with embedding $e_q$ as

$$s(d \mid q) = \sum_{k} p(k \mid e_q)\,(1-a_k)\Big[\log \mathcal{N}(e_d;\mu_k,\Sigma_k) - \lambda\,(e_d-e_q)^\top \Sigma_k^{-1}(e_d-e_q)\Big].$$

Here $p(k \mid e_q)$ is the query's posterior membership in topic $k$ under the adapted Gaussians, and $\mu_k$ and $\Sigma_k$ are that topic's center and covariance. The first bracketed term rewards demonstrations near the center of the topic. The second is a query-to-demonstration distance measured in the topic's own geometry, so that movement along the stretched lay-to-expert axis is cheap. $\lambda$ trades the two off.

The prototypicality term is our response to the finding in Margatina et al. that feeding the model examples it finds confusing hurts in-context learning. We target topics where the model is weak, but within them we pick central examples, not boundary cases. Final selection is greedy, with a coverage penalty on topics already represented, following TopicK.

Baselines

The naive reference points are random demonstrations and BM25, the classic term-matching ranker. Both run on the same pool with the same number of demonstrations (4 and 8) and the same target model, an open 8B-class instruction model such as the Llama-3.1 8B used in PQR, with fixed prompt templates and three orderings per query.

The standard semantic baseline is cosine nearest-neighbor retrieval over the same encoder. This is the KATE setup that Margatina et al. found dominant over random, diversity, and uncertainty selection. We also run it with a stronger encoder (BGE-m3), since a better encoder is the cheapest fix. The same retrieval with globally whitened embeddings is a further control: it tests whether a single corpus-wide reshaping gets the benefit without topic-specific shapes.

The strongest competitor on this exact question is TopicK. It already targets knowledge gaps and already wins on MedMCQA and Law. We reimplement it on our encoder and pool, with the same number of topics and the same competence-probing budget. The file reports its gain of up to 6.38% on these domains, but no reported result found on Lacuna for TopicK's absolute MedMCQA or Law accuracy, so we will rerun it ourselves rather than compare against a quoted number.

As the strongest trained retriever we include UDR. It is a dense retriever trained on feedback from the target LLM, reported as beating BM25 and SBERT by 10 points on average on unseen tasks, and two of its demonstrations beat eight from BM25. We train it with its published recipe on the same pool and match our tuning budget.

For the lexical gap we include two generation-based baselines. The first is LLM query rewriting: the model restates the lay query in expert register, and cosine retrieval follows. This is in the spirit of RaFe and InteR. The second is PQR-style expansion, which generates potential queries per pool example and scores by the maximum over mixture means. For reference, PQR reports raising MiniLM's BEIR average NDCG@10 from 0.420 to 0.464, though that is passage retrieval rather than demonstration selection.

Three ablations isolate the proposal's ingredients. The first replaces full low-rank covariances with spherical ones, which reduces our method to soft-assignment TopicK. The second removes the transductive covariance update. The third removes the prototypicality term.

Beating the field means two things. We must exceed TopicK and query rewriting in accuracy on the mismatch splits, with non-overlapping bootstrap confidence intervals over test questions. We must also stay within a few tenths of a point of cosine retrieval on the matched splits.

Why not just…?

Why not just use a stronger dense encoder with cosine nearest neighbors? BEIR found that dense retrievers trained on general web data often fall below BM25 on specialized corpora with their own vocabulary. That suggests encoder strength alone does not remove register shift. Still, BGE-m3 is far stronger than the models BEIR tested. We run it directly: if BGE-m3 cosine retrieval matches us on the lay-register splits, the covariance mechanism is unnecessary, and we will say so.

Why not just rewrite the query into expert language with an LLM? This is the real threat. Rewriting attacks the mismatch head-on and may well match our accuracy. Its known costs are one generation per query at inference and the risk of drift or hallucinated constraints. We measure both accuracy and rewrite fidelity, judged against the paired original expert question. We also test rewriting combined with our retriever. If rewriting wins outright, our contribution shrinks to a zero-generation alternative with a stated accuracy gap, and the note will report it that way.

Why not just use TopicK with soft Gaussian clusters? The spherical-covariance ablation is exactly this. If it matches the full model, the covariance is doing no work, and the idea reduces to a minor smoothing of TopicK. The paired-consistency diagnostic below shows directly whether topic shape, not softness, is what keeps lay and expert versions of a question in the same topic.

Why not just generate potential queries for each pool example, as PQR does? It may work, but it costs hundreds of LLM generations per example and must be redone whenever the pool changes. It also still scores against means only. We run a reduced-budget version, 30 generated queries per example, and report accuracy per GPU-hour against our generation-free method.

Proposed Experiments

The main testbed is TopicK's suite: MedMCQA, Law, SciQ, CommonsenseQA, QNLI, and MNLI. Medical and legal tasks are where TopicK saw its largest gains, and where lay and expert phrasing diverge most. For MedMCQA and Law we build a controlled mismatch split. An LLM rewrites each test question into layperson register while keeping the answer fixed; clinicians or law students verify a subsample for answer preservation. The pool stays in the original expert register.

Because we have paired lay and expert versions of each question, we can measure topic-assignment consistency directly: the probability that both versions receive the same most-likely topic. We report this for our adapted Gaussians, the unadapted ones, TopicK's classifier, and cosine nearest neighbors (top-1 neighbor's topic).

We also report downstream accuracy binned by content-word overlap between each query and its retrieved demonstrations. The prediction is specific: our advantage over cosine retrieval and TopicK should widen in the lowest-overlap bins and vanish in the highest.

A component check on Web Snippets tests whether the adapted topic Gaussians, used alone as a classifier, reach Ma et al.'s reported 85.48% accuracy with static 400-dimensional embeddings. Falling short of that would mean our contextual-encoder Gaussians are worse density models than their static ones, and we would revisit the encoder choice before trusting the retrieval results.

One known risk comes from Gaussian LDA. Its authors observed that embedding-space Gaussians miss topics defined purely by which words co-occur in documents. We therefore inspect topics where our method underperforms TopicK for this signature.

The decisive result is concrete. On the lay-register MedMCQA and Law splits, the full model must beat TopicK and cosine retrieval, and the spherical and non-adapted ablations must fall back toward TopicK. That pattern would show the query-adapted covariance, not topics or softness alone, is what closes the gap. Any other pattern would falsify the core claim.