A voice assistant cannot show a ranked list. It retrieves documents, picks one short sentence, and says it aloud. The listener has no screen to look back at and no second result to compare against. That listener is also poor at noticing when the spoken sentence is fluent but wrong. Current methods for teaching a language model to rewrite and expand search queries ignore this. They are trained with reinforcement learning (RL) to push the correct document somewhere into the top 10 or top 100 results, which is the right target for a screen and the wrong one for an ear. We propose to train a small query-expansion model with a reward that scores only what a listener actually hears. The reward is positive when the single spoken sentence is correct, zero when the system declines to answer, and negative when it speaks confidently and wrongly.
The motivation comes from human studies of spoken question answering. When answers are delivered by voice, short extractive sentences let listeners catch 75.6% of system errors. Long paragraphs drop that to 70.4%, and abstractive summaries to 71.3%. Listeners also prefer bare answers that make their errors hardest to catch (Do Explanations Help Users Detect Errors in Open-Domain QA? An Evaluation of Spoken vs. Visual Explanations). In speech, then, a wrong but plausible top answer is costly and mostly invisible to the user. The system that decides which documents get retrieved, the query expander, should be trained against that cost directly.
Core Claim
Generative query expansion means having a language model write extra text before search, such as a rewritten query or a hypothetical answer, so the search engine can find documents that use different words than the user did. For voice-only conversational search, we claim this expansion should be optimized for top-1 spoken correctness under an asymmetric penalty for confident errors, not for rank-based retrieval metrics. We expect the two objectives to diverge in a measurable way. Policies trained on rank rewards should raise Recall@10 and MRR but produce more "confidently wrong" spoken answers. These errors should cluster at topic switches and in turns where the generated hypothetical answer invents entities. A policy trained on the spoken reward should give a better risk–coverage curve, meaning higher accuracy on the turns it chooses to answer, at any fixed answer rate.
The research file suggests a more obvious idea: add a length or listenability penalty to the expander. We think that idea is aimed at the wrong target. CONQRR showed that RL rewriters learn long, redundant queries. It also treated those queries as internal payloads that are never shown or spoken. Shortening them buys the listener nothing and likely costs recall. The auditory constraint enters through what the system chooses to say, and whether it says anything at all, not through the length of the internal query. We keep the length penalty only as an ablation meant to confirm this.
Relation to Prior Work
Three conversational search systems supply our methods and set the baselines. ConvSearch-R1 trains a 3B rewriter in two stages. First, supervised fine-tuning (SFT) on its own zero-shot rewrites that already retrieve the gold passage. Second, GRPO (group relative policy optimization, an RL method that compares several sampled outputs per prompt), with an explicit reasoning step before the rewrite. Its reward, Rank-Incentive Reward Shaping, gives partial credit for placing the gold passage in the top 10 or top 100. We reuse its two-stage recipe and swap the reward. Its partial credit for rank 50 is exactly what a listener never experiences.
ConvGQR appends a generated hypothetical answer to a rewritten query and aligns representations with a supervised loss. That paper, together with HyDE, motivates using a hypothetical answer as a search pivot. The same papers name the risk we target: hallucinated pivots cause topic drift. Neither uses RL, and neither measures what happens to the spoken answer.
Aligned Query Expansion is the closest precedent for a top-1 target. It fine-tunes a 3B model with rejection sampling and DPO (direct preference optimization, learning from ranked pairs of outputs) according to the BM25 rank of the gold document. It reports 30.8% top-1 accuracy on Natural Questions, against 22.1% without expansion. AQE is single-turn, rewards passage rank rather than sentence correctness, and has no abstain option or error asymmetry. We differ on all three.
Two further lines of work shape the proposal. The calibration role of RL in CARE: Confidence-Aware Reasoning for Reliable Medical VQA is the precedent for training a policy to back off rather than assert. On the auditory side, Conversational Information Seeking: Theory and Application argues that voice-only search cannot present lists and that offline metrics such as nDCG ignore listening constraints. The spoken-versus-visual study above supplies both our evaluation unit, the single extractive sentence, and our human protocol.
Domain specialization plays a supporting role here. The goal is a compact 1.5B–3B model that fits real-time voice latency, following the evidence that small task-specialized models can beat larger generalists (ConvSearch-R1; Lightweight Transformers for Conversational AI).
Research Program
The system has three parts. The first is a trainable expander. Given the dialogue history, it writes a short reasoning trace, a rewritten query, and a brief hypothetical answer, then emits a speak-or-abstain token. The second is a frozen retriever: BM25, and separately a dense dual encoder. The third is a frozen sentence selector, an off-the-shelf cross-encoder that picks the single best sentence from the top-ranked passage. The selector scores sentences against the rewritten query and the hypothetical answer together.
If the policy speaks, the spoken output is that one sentence. If it abstains, the output is a fixed short prompt asking the user to rephrase.
The reward for one turn is written as follows.
$$R = \begin{cases} 1 & \text{speak and } s \text{ contains the gold answer} \\ -\lambda & \text{speak and } s \text{ does not} \\ -c & \text{abstain} \end{cases}$$
Here $s$ is the spoken sentence. Containment is judged by token-F1 against the gold answer, above a threshold we fix on development data. The weight $\lambda > 0$ is the extra cost of a confidently wrong spoken answer. The small cost $c \ge 0$ prevents the policy from abstaining on everything.
Sweeping $\lambda$ traces out a family of policies along a risk–coverage curve. Because top-1 sentence correctness is a sparse signal, which is the failure mode flagged for binary rewards in ConvSearch-R1, we add a small shaping term during early training only. It gives credit when the gold passage appears in the top 5. We then anneal it to zero, so the final policy is trained purely on the spoken outcome.
Training follows ConvSearch-R1's two stages. Stage one is SFT on self-generated traces whose spoken sentence was correct. Stage two is GRPO with the reward above. The base models are 1.5B and 3B open models.
To reflect the voice setting on the input side, we build a spoken-query variant of each benchmark. User turns are passed through text-to-speech and then speech recognition, and the model is trained and tested on the noisy transcripts. Spoken queries are short and error-prone, as the Alexa study in Query Expansion and Entity Weighting for Query Reformulation Retrieval in Voice Assistant Systems documents.
Baselines
The naive reference point sends the raw current user turn, with no rewriting or expansion, to the same retriever and sentence selector. It always speaks. This fixes the floor for spoken accuracy and for the confidently-wrong rate.
The strongest existing method is ConvSearch-R1, retrained by us at the same model size, on the same training split and noisy-ASR inputs, with the same GRPO compute budget, the same number of sampled rollouts per prompt, and the same tuning budget. It is fed through the identical sentence selector. To make its comparison fair on the abstention axis, we add a post-hoc abstention threshold on the selector's score, tuned on development data.
For published figures: ConvSearch-R1 reports MRR@3 gains of over 10% on TopiOCQA relative to earlier methods. The file gives only the relative gain; no absolute ConvSearch-R1 score was found on Lacuna. Where the file gives absolute numbers, we compare retrieval metrics to them. ConvGQR reports dense-retrieval MRR of 42.0 on QReCC, against 38.4 for human rewrites, and 46.5 MRR zero-shot on CAsT-20. AQE reports 30.8% top-1 accuracy on Natural Questions. No reported result was found on Lacuna for any system's top-1 spoken-sentence accuracy or confidently-wrong rate on these benchmarks, so we establish those numbers ourselves for every baseline.
A second strong competitor is AQE-style training adapted to conversation. We sample expansions, rank them by the gold passage's top-1 rank, and fine-tune with rejection sampling plus DPO. This isolates whether an existing top-1 passage objective already captures what we want.
We also run ConvGQR's rewrite-plus-hypothetical-answer concatenation as the non-RL generative expansion baseline. A prompted frontier model serves as a latency-unconstrained reference. It is told to rewrite, expand, and decide whether it is confident enough to answer.
Two ablations isolate the key ingredient. The first keeps our full pipeline but replaces the spoken reward with Rank-Incentive Reward Shaping, changing only the reward. The second keeps the spoken reward but sets $\lambda = 0$ and removes the abstain action. This separates the gain from choosing the right target (top-1 sentence) from the gain from the error asymmetry. A third ablation adds a query-length penalty to ConvSearch-R1, to test and we expect retire the "shorten the query" idea.
Beating the strongest baseline means one thing. At matched coverage (the fraction of turns answered, at 80% and 90%), our policy must reach higher selective accuracy on the spoken sentence and a lower confidently-wrong rate than ConvSearch-R1 with its tuned threshold. The margin must exceed seed-to-seed variation across three training seeds, on both TopiOCQA and QReCC, with the direction holding zero-shot on CAsT-20.
Why not just…?
Why not just keep the best retrieval rewriter and add a confidence threshold on the sentence selector? This is the serious threat. The file's case against decoupled pipelines rests on abstractive summarization failing in speech, and that does not apply to extractive selection. A post-hoc threshold can only filter what the rewriter already retrieved. It cannot teach the rewriter to avoid invented entities in its hypothetical answer, which is what produces plausible wrong passages at rank 1. So our prediction is specific. At matched coverage, a threshold will trade away correct answers to remove wrong ones, while the trained policy shifts which documents appear at rank 1. The head-to-head risk–coverage comparison settles this directly. If the curves overlap within seed variance, we conclude that decoupling suffices and report that as the finding. The remaining contribution would then be the spoken-sentence evaluation protocol.
Why not just penalize the length of the expanded query, since listeners need brevity? The expanded query is never spoken. CONQRR's redundant rewrites are machine payloads, and brevity matters only for the uttered sentence, which the extractive selector already keeps short. We run the length-penalized ConvSearch-R1 ablation. We expect it to lower retrieval recall without improving spoken accuracy or the confidently-wrong rate. If it did improve them, our reading of where the auditory constraint enters would be wrong.
Why not just prompt a frontier model to rewrite, expand, and decide whether to answer? Closed-book GPT-3 reached only 31.8 F1 on TopiOCQA, against 55.8 for a retriever–reader system (TopiOCQA: Open-domain Conversational Question Answering with Topic Switching). AQE also cut compute time by 69.5% by replacing test-time sampling with a trained small model. Neither result measures frontier models acting as expanders with abstention, though, so we include the frontier model as a measured reference rather than dismiss it. It may win on accuracy. If it does, the question becomes whether a 3B model closes the gap at voice-compatible latency, which we report in milliseconds per turn.
Why not just use rejection-sampled SFT on expansions that retrieve correctly, without RL? This is ConvSearch-R1's first stage and AQE's rejection-sampling step. It is cheap, which matters because RL took roughly 4.5 times longer to train than supervised baselines in CONQRR. It cannot learn when to abstain from positive examples alone, because the harm of a confident error only appears through a negative reward. We report stage-one-only results at each model size, so the added value of RL is measured rather than assumed.
Proposed Experiments and Decisive Checks
The main experiment trains 1.5B and 3B expanders on TopiOCQA and QReCC under both clean and ASR-noisy user turns, with BM25 and dense retrievers. We evaluate top-1 spoken-sentence accuracy, selective accuracy at 80% and 90% coverage, area under the risk–coverage curve, and confidently-wrong rate. We also report MRR@3 and Recall@10, so the retrieval cost of the new objective is visible. CAsT-20 serves as a zero-shot transfer test.
The first decisive check is divergence. If rank-reward and spoken-reward policies make nearly identical top-1 sentence choices, the premise fails. We will see this early from the reward-swap ablation.
The second check is localization. Using TopiOCQA's topic-switch annotations and an entity-overlap test between each hypothetical answer and the dialogue history, we test whether the reduction in confidently-wrong answers concentrates at topic switches and hallucinated pivots, as the plausibility-trap account predicts.
The third check is whether automatic gains reach listeners. We run a listening study with the protocol of the spoken-versus-visual study. Participants hear the spoken sentence from each system on the same turns, played by text-to-speech, and judge whether the answer is correct. Our target sample is about 150 participants, each hearing a fixed set of turns per system, with the size to be confirmed by a power analysis before the study begins. The outcomes are error-detection accuracy and end-task accuracy, which counts a turn as correct if the user accepted a correct answer or rejected a wrong one. The published spoken EXT-SENT figure of 75.6% is a reference point for error detection. Our claim is about the product of what the system says and what the listener catches. The proposal succeeds only if the spoken-reward policy improves end-task accuracy over the thresholded ConvSearch-R1 baseline at matched coverage.
References
- Do Explanations Help Users Detect Errors in Open-Domain QA? An Evaluation of Spoken vs. Visual Explanations
- CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning
- ConvSearch-R1: Enhancing Query Reformulation for Conversational Search with Reasoning via Reinforcement Learning
- ConvGQR: Generative Query Reformulation for Conversational Search
- Precise Zero-Shot Dense Retrieval without Relevance Labels
- Aligned Query Expansion: Efficient Query Expansion for Information Retrieval through LLM Alignment
- CARE: Confidence-Aware Reasoning for Reliable Medical VQA
- Conversational Information Seeking: Theory and Application
- Lightweight Transformers for Conversational AI
- Query Expansion and Entity Weighting for Query Reformulation Retrieval in Voice Assistant Systems
- TopiOCQA: Open-domain Conversational Question Answering with Topic Switching