Carrying Expert Hypotheses Across Search Turns with a Small Model

When a tenant types "water keeps coming through my ceiling," then "landlord says it's not his problem," then "can I stop paying rent," the documents that answer the third question are statutes about the implied warranty of habitability and rent withholding. None of those words appear in anything the user typed. Search systems handle this badly for two separate reasons. They usually treat each query as a fresh, isolated request. And even when they see the whole conversation, in expert domains the user's words and the relevant documents barely overlap. In legal retrieval, query–document word overlap is about a third of what it is in open-domain question answering, and keyword search finds the right statute in the top ten only about 5% of the time (A Reasoning-Focused Legal Retrieval Benchmark). The fix that works today is expensive. A large model reasons about the full conversation before every search. On a multi-turn, reasoning-heavy benchmark, adding history plus explicit reasoning roughly doubles retrieval quality (RECOR), but paying for that call on every turn is impractical in an interactive system.

We propose a small language model (3B–8B parameters) that keeps a short written list of the domain concepts it currently believes are in play, which we call a hypothesis ledger. Examples are a legal doctrine, a candidate diagnosis, or a governing theorem. At each turn the model updates the list, drops entries the user has moved away from, and writes a standalone search query from the current question plus the surviving entries. The model is trained not to copy a teacher's rewrites but to produce queries that actually retrieve the gold documents, with retrieval quality as the reward. The bet is twofold. First, a compact, explicit, revisable state attacks the failure RECOR identifies, where 63% of retrieval failures fall in later turns because the logical link to the gold document is never stated. Second, rewarding retrieval rather than imitation is what lets a small model match a large one here.

Core Claim

A specialized small rewriter whose only memory of the session is a bounded ledger of domain hypotheses, trained with reinforcement learning against a retrieval reward, can recover most of the gain from per-turn large-model reasoning on multi-turn expert retrieval. It should do so at a fixed, small per-turn cost. The gain should sit specifically in late turns and in sessions where the user changes topic.

What would be new, measured against the closest work, is the following. Enhancing Conversational Search: LLM-Aided Informative Query Rewriting distills ChatGPT's "informative" rewrites into T5. It works on open-domain conversation, where good rewriting means resolving pronouns and pulling forward earlier phrases, and it learns by imitation. BRIGHT and the legal benchmark above show that having a large model reason before retrieval helps by up to 12.2 and roughly 6–9 points respectively, but only for single isolated queries. RECOR measures the multi-turn version of this but proposes no method. Our proposal differs from all three in two ways. The state carried between turns is a set of inferred expert concepts rather than raw history or surface tokens. And the supervision is retrieval outcome rather than teacher text.

Research Program

The model receives the current user turn, the previous ledger, and the titles of the top few documents it retrieved on the previous turn. It emits an updated ledger and then a rewritten query. The ledger holds at most $K$ entries; we start with $K=6$. Each entry is a short domain concept with a status of active or dropped and a pointer to the turn that introduced it. Because the model never rereads the full transcript, its input length stays roughly constant as sessions grow.

This design responds directly to two findings from session search. In the NTCIR-16 partially observed session subtask, blindly feeding the previous three queries made ranking worse than using no history at all: the session-level ranking score fell from 0.423 to 0.379 (MM6 at the NTCIR-16 Session Search Task). Separately, models given history can learn to ignore the current query (Query-oriented Data Augmentation for Session Search). Explicitly dropping hypotheses is our answer to the first finding. Always building the rewrite around the current turn is our answer to the second.

Training has two stages. The first is a supervised warm start. A large teacher model reads synthetic sessions and writes ledgers and rewrites, which the small model learns to reproduce, following the distillation recipe of Specializing Smaller Language Models towards Multi-Step Reasoning. The second stage is reinforcement learning with a group-relative policy optimizer, the same algorithm used for verifiable-reward training in AutoTriton. Here the reward is how well a fixed, frozen retriever does with the rewrite:

$$r_t = \tfrac{1}{2}\left[\mathrm{nDCG@10}_{\text{BM25}}(\hat q_t) + \mathrm{nDCG@10}_{\text{dense}}(\hat q_t)\right] - \lambda\,|\hat q_t|$$

In this formula, $\hat q_t$ is the rewrite at turn $t$, nDCG@10 is the standard graded ranking score over the top ten results computed against the session's gold documents, the subscripts name a keyword retriever and a dense retriever, $|\hat q_t|$ is the rewrite length in tokens, and $\lambda$ is a small penalty weight. Averaging over two retrievers discourages rewrites tuned to one retriever's quirks. The length penalty stops the model from padding the query with every concept it knows.

Training sessions are built with RECOR's decomposition-and-verification recipe. A complex answer is split into atomic facts, and each fact becomes one conversational turn whose gold documents are known. We apply the recipe to domain corpora disjoint from all test sets: StackExchange-style corpora in the BRIGHT domains, U.S. statutes, and medical guidelines. During training, we splice in unrelated turns from other sessions, so the model must learn both when to drop entries and when to carry them forward.

Baselines

The naive reference point is stateless retrieval on the raw current turn with BM25 and a strong dense encoder. RECOR reports 0.236 nDCG@10 for its no-history baseline, and we will reproduce that setting.

The second baseline is plain concatenation of prior turns into the query, in the style of THUIR2's assembled session queries (THUIR2 at NTCIR-16). It costs nothing to run and is known to sometimes hurt.

The strongest method on cost-blind quality is RECOR's history-plus-reasoning setting, where a large model reasons over the full conversation before each retrieval. RECOR reports 0.479 nDCG@10 for it. The strongest retriever on the benchmark is the reasoning-specialized dense model DIVER, which RECOR reports at 0.584 nDCG@10, the highest individual score in the paper. The file does not say which query input DIVER was given, so we will rerun DIVER both on raw history and on our rewrites.

We will run every condition on the same corpora, the same retrievers, and the same test turns. Rewriters of comparable size get the same GPU-hour budget for tuning. The large-model baseline is reported both at full cost and at matched latency.

Beating the field means three things. Our rewrites fed to BM25 and a standard dense encoder should reach at least 90% of the 0.479 history-plus-reasoning figure. Our rewrites fed to DIVER should exceed 0.584. And both should hold at under a tenth of the large model's per-turn generation cost.

We also include CORAL's large-model summarization of the history (CORAL), which gave the best grounding in late turns on Wikipedia conversations.

Two ablations isolate the key ingredients. The first keeps the same small model and training data but has it rewrite from the full transcript without a ledger. The second keeps the ledger but trains only by imitating the teacher, with no retrieval reward.

Why not just…?

Why not just call a frontier model with the full history on every turn? It is the quality ceiling, and we treat it as one rather than pretending to beat it. The case against it as the deployed system rests on cost and late-turn noise. CORAL found that grounding degrades over turns 4–6 when raw history is passed in. Compact specialized models have also beaten larger generalists when trained for the target task: an 8B clinical model outperformed Gemini-1.5-Flash in EC-RAFT, and a fine-tuned 1.5B model with query rewriting beat GPT-3.5 at conversational retrieval in CORAL. If our model cannot reach about 90% of this baseline, the claim fails.

Why not just distill teacher rewrites into a small model, as Ye et al. did? This is the real threat. It may be most of the gain. Imitation inherits the teacher's blind spots, and the teacher has no signal about which concepts actually retrieve the gold documents. The imitation-only ablation settles the question directly. If adding the retrieval reward does not improve late-turn nDCG@10 by a clear margin over imitation alone, we will report that the contribution reduces to the ledger format plus distillation.

Why not just use a reasoning-trained retriever like DIVER on the concatenated history and skip rewriting? RECOR shows such retrievers are strong, so this is also a serious competitor. However, a retriever that has to embed an ever-growing transcript faces the same problem MM6 found, where stale context drowns out the current need. We test the two options both separately and stacked. If the rewriter adds nothing on top of DIVER, the rewriter is only useful for cheaper retrievers, and we will say so.

Why not just summarize the history? CORAL's summaries compress the flow of the conversation, but they keep the user's own words. In the lexical desert of legal and medical retrieval, the missing piece is the expert concept itself, and a summary never introduces it. The comparison on sessionized legal data, described below, tests this directly.

Proposed Experiments

The primary test set is RECOR, held out entirely from training. We report nDCG@10 overall and broken down by turn position, with turns three and later reported separately because that is where 63% of RECOR's failures fall. We also report results by domain, with law and biology highlighted.

The second test set consists of sessionized versions of Bar Exam QA and Housing Statute QA (A Reasoning-Focused Legal Retrieval Benchmark) and of R2MED. In each, a single fact pattern is revealed across turns. Single-shot reference points exist for these: 5.03% BM25 Recall@10 on Bar Exam QA, and a 41.4 nDCG@10 ceiling for large-reasoning-model pipelines on R2MED. These let us check that the multi-turn construction still produces meaningful retrieval.

A drift test splices off-topic turns into sessions and measures two things. One is whether ledger entries are dropped at the switch. The other is whether retrieval on returning turns recovers to its pre-switch level. We expect a full-history rewriter to lose more than the ledger model at these points.

To measure the specialization tax that Specializing Smaller Language Models towards Multi-Step Reasoning observed on BigBench Hard, we evaluate the trained rewriter on open-domain QReCC rewriting. We compare BM25 MRR there against the 49.39 informative-rewrite figure and the 39.81 human-rewrite figure from Ye et al.

Latency and generated tokens per turn are logged for every system.

Three results would decide the question. The first is the gap between the full model and the imitation-only ablation on late turns. The second is the gap between the ledger and the full-transcript rewriter on drift sessions. The third is whether our rewrites push DIVER above 0.584 on RECOR. If none of these gaps appears, the honest conclusion is that per-turn distillation already captures the effect, and the ledger and reward add nothing worth their complexity.