When people search through a conversation, they write "does it come in blue?" or "what else did he make?" and expect the system to know what "it" and "he" mean. Conversational query rewriting handles this. A language model reads the dialogue and turns the latest utterance into a standalone search query that an ordinary search engine can answer. Published rewriters now score well on public benchmarks such as TopiOCQA and QReCC. Those scores cannot tell apart three possible causes of failure, and they cannot rule out a fourth explanation for success. A rewriter can fail because it was distracted by irrelevant earlier turns. It can fail because the thing being referred to was shown in an image rather than named in text. It can fail because the user jumped back to an old topic. Or it can succeed only because the benchmark's gold rewrites were in its pretraining data. This proposal builds a generator of conversations in which the referent of every ambiguous turn is known exactly: which earlier turn introduced it, whether as text or image, and how much distracting material lies between. Because the entities and the documents to be retrieved are freshly invented, memorization cannot help.
The generator does two jobs. As an evaluation instrument, it lets us split a rewriter's errors into distraction, cross-modal binding, topic-return, and memorization components. As a training signal, it gives free, exact labels for "where the answer to this reference lives." We use those labels to test one specific architectural question. Can a sparse mixture-of-experts rewriter fix distraction in a single forward pass? Such a model has several specialized sub-networks plus a small gating network that picks which ones process each token. The question is whether supervising its gate with these labels lets it match the slower multi-stage pipelines that currently handle noise. The design is built so that a cheaper dense alternative could win, and we report it if it does.
Core Claim
The claim has two parts. First, part of the reported progress of LLM-based rewriters on TopiOCQA and QReCC reflects memorization. This part is measurable as the drop in retrieval quality when the same conversations are re-instantiated with substituted entities and a matching substituted corpus. Second, once memorization is removed, the dominant remaining failures are distraction by intervening turns and referents carried only by images. Supervising which turn and which modality holds the referent reduces both failures more than supervising the rewrite text alone. The new element is the referent-level supervision, which static benchmarks cannot provide. The sparse routing is the vehicle we test for using it cheaply, not the claim itself.
Grounding in Prior Work
Several papers motivate the gap by showing that conversational history acts as noise. TopiOCQA builds open-domain conversations in which 28% of turns switch topic. It finds that models fail on "switch-to-old" turns because recent history pulls them toward the wrong subject, and it leaves a 14.2-point F1 gap to humans (55.8 vs 70.0). CHIQ shows that 7B open models fail at end-to-end rewriting because of messy context. It fixes this with five sequential history-cleaning subtasks before rewriting, reaching MRR 38.0 on TopiOCQA and 47.2 on QReCC. We keep CHIQ's diagnosis that distraction is the problem but question its remedy. CHIQ is multi-stage, text-only, and evaluated only on public data.
On the text-only open-domain QA benchmarks that Cho et al. study, reader accuracy collapses as more irrelevant documents are added. That supports treating distraction as a property of soft attention rather than of any one dataset. Fix Before Search shows the visual counterpart. Cluttered image queries lose 98.6% of retrieval recall, and rotation alone costs 49% of Recall@5. Their agentic repair loop restores it, but at 109.4% relative latency, and only for single-turn queries.
Two papers supply methods we reuse. AdaCQR argues that human gold rewrites are not the best retrieval queries. It aligns a T5 rewriter to fused BM25 and dense retrieval ranks, improving QReCC sparse MRR from 33.4 (supervised T5QR) to 52.4. We adopt its retrieval-aligned objective as the base loss for every trained system, so that differences come from referent supervision and not from objective misalignment. I2MoE trains experts specialized for unique, redundant, and synergistic multimodal information. It uses weak supervision obtained by masking modalities, which gave gains such as +5.5% on ADNI over a dense MulT baseline. We replace its weak, perturbation-derived signal with exact labels from the generator, and we move the idea from static classification to multi-turn rewriting.
JDDC 2.1 constrains the multimodal regime. It is the only multimodal query-rewriting dataset in the file, and its manual audit finds that 74% of customer-service responses need the images. It also uses dense ResNet-plus-GPT fusion, scored only by BLEU (5.45 multimodal vs 2.38 text-only). That makes it evidence that images matter, but not a retrieval benchmark.
The evaluation design follows DyVal. DyVal generates reasoning problems from dependency graphs with controlled depth and finds that models strong on static benchmarks score near zero on fresh instances. Its authors note that this approach has stayed inside arithmetic and logic. We carry it to dialogue: the conversation's coreference structure is the graph.
Research Program
The generator starts from a synthetic world: a set of invented entities (products, people, places) with attributes. Each entity has a small set of generated passages that form a closed retrieval corpus of a few hundred thousand passages. Some entities are bound to images. These images are drawn from an interleaved image–text pool such as PIN-14M, and the invented names and attributes are attached to them. Memorizing an image therefore says nothing about the fictional name the conversation binds to it.
A conversation is sampled as a graph. Each user turn either introduces an entity, by name or by image only, or refers back to one by pronoun, ellipsis, or an image re-shown. The generator records the source turn and modality of every referent, the number of intervening distractor turns, and the topic-switch type (none, switch-to-new, switch-to-old). It also records whether an assistant turn plants a plausible but wrong entity. That last setting is the hallucination bait that CHIQ's pseudo-response module is exposed to. Visual clutter is applied with the degradation families from Fix Before Search.
An LLM paraphrases the templated turns into natural language. We filter the paraphrases with an entailment check so that the referent structure is preserved. The gold standalone query is defined by the graph rather than by an annotator. Retrieval success is measured directly against the closed corpus.
A second, smaller component re-instantiates real conversations. We substitute every named entity in TopiOCQA and QReCC test dialogues with an invented counterpart, and we rewrite the gold passages to match, keeping their discourse structure intact. The memorization gap for a system is its retrieval MRR on the originals minus its MRR on these twins. This gap is the most direct contamination measure available for conversational search. The file notes that no such measure exists for this task.
The trained model is a vision-language rewriter. Its gating network receives an auxiliary loss asking it to send tokens of the referent-holding turn (and modality) to a designated "anchor" expert group, and other history tokens elsewhere. The model is trained on generated data plus the original training splits, under the AdaCQR retrieval-aligned objective. The point is a single pass with no separate cleaning stage.
Baselines
The naive reference is raw concatenation: the full dialogue history plus image embeddings goes straight to the retriever with no rewriting. Sekulić et al. report that this loses to generative rewriting. It anchors the scale and shows how hard each noise level is. We also run "current utterance only" as a floor.
The supervised standard is T5QR, a T5 model fine-tuned on human gold rewrites. AdaCQR reports it at QReCC sparse MRR 33.4. We retrain it on the same training conversations as our model.
The strongest existing text methods are CHIQ-Fusion and AdaCQR. CHIQ-Fusion reports TopiOCQA MRR 38.0 and QReCC MRR 47.2. AdaCQR reports QReCC sparse MRR 52.4 and TopiOCQA dense MRR 32.8. These were measured under different retrieval setups and are not comparable to each other. We therefore rerun both with identical BM25 and dense retrievers, identical training data where training applies, and the same hyperparameter-search budget in GPU-hours as our model.
For image-bearing turns, CHIQ and AdaCQR receive a caption from the same vision encoder, which is the most charitable text-only adaptation. We also chain Fix Before Search-style image repair in front of them, with its latency counted. JDDC 2.1's dense multimodal GPT has a reported BLEU but no retrieval number. No reported result was found on Lacuna for any multimodal rewriter on a retrieval metric, so we reimplement dense fusion at matched size. A prompted frontier model (GPT-4o, zero-shot and few-shot) is included because it is what a practitioner would try first.
Beating the field means two things. On generated data and entity-swapped twins, the model must exceed the rerun CHIQ-Fusion and AdaCQR in MRR on the switch-to-old, long-distractor, and image-referent slices. It must also stay within one MRR point of them on the original benchmarks, and do this at lower wall-clock latency than CHIQ's pipeline.
The ablations isolate the key ingredient. The first is the same sparse model with unsupervised routing. The second, and most important, is a dense model of matched active parameters with the same referent labels used as an auxiliary token-relevance head instead of routing supervision. The third is the full model trained without generated data.
Why not just…?
Why not just prompt GPT-4o with the whole conversation? It may do well on public benchmarks for exactly the reason we want to measure. A static score cannot separate reasoning from recall. DyVal reports a 13.6% drop for GPT-4 when only the description order changes. Fix Before Search found that GPT-4o was matched by a fine-tuned 4B model on visual repair. Our twin experiment answers this directly: if GPT-4o's memorization gap is near zero and it wins on generated data, the case for a trained rewriter weakens, and we will report that.
Why not just run CHIQ's history cleaning, with captions for images? This is a serious competitor on text-only distraction. On image-only referents, however, a caption must name the entity, and the name is bound only by the conversation. The cost is also real: CHIQ is multi-stage, and adding image repair costs 109.4% latency according to Fix Before Search. The image-referent slice and the latency measurement settle whether a single pass can match it.
Why not just train a dense vision-language rewriter with the same referent labels? This is the main threat to the architectural half of the claim. The labels, not the sparsity, may be what helps. The dense-with-relevance-head ablation is designed to show this. If it matches the sparse model at equal active compute, we will conclude that referent supervision is the contribution and that mixture-of-experts adds nothing. The generator and the memorization measurement still stand on their own.
Why not just evaluate on post-cutoff conversations instead of synthetic ones? Fresh real conversations would be ideal for contamination, but they come without referent labels and without a matched corpus. They also cannot vary distractor distance or topic-return type independently. We use entity-swapped real conversations as the bridge between the two.
Proposed Experiments
The first decisive check is the memorization gap. It is computed for every baseline on entity-swapped TopiOCQA and QReCC, under both BM25 and dense retrieval. A gap well above the run-to-run variance for LLM rewriters, and a smaller one for retrieval-aligned trained rewriters, would confirm the contamination half of the claim.
The second check is validity. Generated benchmarks are only useful if they rank systems the way real ones do once contamination is removed. We require a Spearman rank correlation between system orderings on generated data and on the entity-swapped twins. If that correlation is weak, the generator's conversations are unrealistic and the paraphrase stage must be revised before any architectural conclusion is drawn.
The third check is the failure decomposition. We plot MRR against distractor distance, topic-switch type, referent modality, and visual degradation, for each system. The question is whether text distraction and visual clutter compound, which no prior work has measured.
The fourth check is the architectural test. Sparse referent-routed, sparse unsupervised, and dense referent-supervised models at matched active parameters are compared on the hard slices. We also inspect whether the anchor experts actually receive the referent turn's tokens on held-out conversations, including real twins where labels come from the substitution map.
Latency per turn is reported for all systems on the same hardware.
References
- TopiOCQA: Open-domain Conversational Question Answering with Topic Switching
- CHIQ: Contextual History Enhancement for Improving Query Rewriting in Conversational Search
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- Fix Before Search: Benchmarking Agentic Visual Query Pre-processing in Multimodal Retrieval-augmented Generation
- AdaCQR: Enhancing Query Reformulation for Conversational Search via Sparse and Dense Retrieval Alignment
- I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts
- JDDC 2.1: A Multimodal Chinese Dialogue Dataset with Joint Tasks of Query Rewriting, Response Generation, Discourse Parsing, and Summarization
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Extending the Use of Previous Relevant Utterances for Response Ranking in Conversational Search