Keeping the Bridge: Training Multi-Hop Retrieval to Survive Speech-Recognition Errors

Many questions need two or more lookups: to answer "In what city was the director of Inception born?", a system must first find the director and then search again for that person's birthplace. The name found in the first step, the bridge entity, becomes the search query for the second step. When the documents being searched are transcripts produced by automatic speech recognition (ASR), such as meeting archives, podcasts, call-center logs, or recorded lectures, the bridge entity is exactly the kind of word the recognizer tends to garble. It mishears a rare surname or turns "Cotton" into "caught in." A single misheard name at hop one then becomes a bad query at hop two, and the whole chain fails. We propose to measure how this failure compounds with the number of hops. We also propose a training method: inject simulated recognition errors specifically at the bridge entities of multi-hop reasoning chains, and teach a step-by-step retrieval model to write a clean, repaired query for the next hop instead of copying the garbled one.

The opportunity comes from three results that have never been put together. Iterative retrieval methods close much of the multi-hop gap on clean text, but they have only been tested on clean text. Studies of transcription noise show that errors on named entities and nouns do the most damage downstream, but those studies cover only single-document tasks. And synthetic error injection has trained noise-robust summarizers and readers, but never a model that has to chain evidence across documents. Our central claim is that in multi-hop retrieval over transcripts, the damage concentrates at the handoff between hops. Correct answers should therefore fall roughly as the per-hop bridge survival rate raised to the number of hops. Putting the injection budget on the bridges during training should recover more accuracy than spreading the same amount of noise uniformly, and more than cleaning transcripts before retrieval.

Where the gap is

Iterative retrieval is the method we build on. IRCoT alternates a chain-of-thought sentence with a retrieval that uses that sentence as the query, and it gains 11–21 points of retrieval recall and up to 15 F1 on HotpotQA, 2WikiMultihopQA, MuSiQue and IIRC. CoRAG fine-tunes Llama-3.1-8B-Instruct on multi-step chains of sub-queries and sub-answers, kept by rejection sampling, which means sampling many chains and keeping only those that reach the correct answer. It raises MuSiQue retrieval recall from 29.0% (single-step) to 47.1%. Both papers note that one bad intermediate retrieval derails everything after it, and both assume pristine text. The mechanism our training method changes is where the next query comes from: in both systems it is generated from text the model has just read.

Two papers motivate the focus on entities. ENDOW runs clean text through speech synthesis, acoustic noise, and Whisper transcription. It finds that generative tasks tolerate roughly 20% word error rate (WER, the fraction of words the recognizer gets wrong) while extractive QA degrades much faster. It also finds that repairing named entities and nouns recovers the most performance, and that repairing verbs recovers almost nothing. Retrieval Augmented Correction of Named Entity Speech Recognition Errors likewise identifies entities as the hardest words to recover. Neither paper looks at chained evidence. The obvious consequence has not been tested: if entities are what break, and entities are what connect hops, multi-hop tasks over transcripts should break far below the 20% single-hop threshold.

The injection methods are borrowed. Spoken SQuAD and Spoken-CoQA build noisy datasets by round-tripping text through speech synthesis and the CMU Sphinx recognizer. MEDSAGE prompts an LLM to corrupt clean text to match a measured error profile of a target recognizer, and gets a 16.4% relative F1 gain in medical summarization on real noisy transcripts. All of these apply noise to a single document and to all words roughly equally. We apply it to the specific tokens that link reasoning steps, and we supervise the model to repair those tokens rather than merely tolerate them.

Two further lines of work constrain the design. Context compressors and pruners such as CompAct, which summarizes retrieved documents step by step and emits a completeness token, and Superposition Prompting, which scores documents in parallel and prunes low-scoring paths, both judge relevance partly by how closely the words match the question. Garbled evidence may therefore be thrown away as irrelevant, so they cannot simply be bolted on. End-to-end audio retrieval (SpeechRAG, WavRAG) avoids transcription altogether. However, WavRAG reports degradation when going from two to three retrieved documents, and SpeechRAG underperforms low-WER text cascades on long audio. That makes them a competitor to test against rather than a solution to the chaining problem.

Core Claim

Let $p$ be the probability that the correct bridge entity survives into the next hop's query, and let $h$ be the number of hops. Suppose each handoff fails roughly independently. Then chain success is bounded by about $p^h$, so a moderate drop in $p$ at fixed WER should hurt 4-hop questions far more than 2-hop ones. ENDOW's single-hop tolerance point would then shrink as $h$ grows. We claim that bridge-targeted training raises $p$ directly: the model learns to produce the canonical entity in its next sub-query even when its evidence contains a misheard version. We also claim this repair beats three alternatives at the same backbone and data budget: cleaning transcripts up front, spreading the same noise uniformly during training, and fuzzy phonetic matching at retrieval time.

Research Program

The first component is a controlled benchmark built from MuSiQue (MuSiQue). We use MuSiQue because its filters remove single-paragraph shortcuts, so an answer requires traversing the bridges, and because it has 2-, 3- and 4-hop questions, which let us test the $p^h$ prediction. HotpotQA and 2WikiMultihopQA serve as secondary sets. We render the paragraph corpus through the ENDOW-style pipeline of speech synthesis, background noise and reverberation, and Whisper transcription, at graded noise levels that span roughly 5% to 40% WER. We keep word-level alignments, so we know exactly which gold bridge mentions were corrupted.

We run two conditions. In the "spoken corpus" condition, questions are typed and the documents are transcripts. In the "spoken everything" condition, the question is transcribed as well. Instrumenting each hop lets us report bridge survival $p$, hop-conditional retrieval recall, and final answer F1 and exact match (EM).

The second component is the training method. We follow CoRAG's recipe of generating chains by rejection sampling with Llama-3.1-8B-Instruct, with one change: while sampling chains, the evidence paragraphs are replaced by versions whose bridge-entity mentions have been corrupted. The corruptions come from round-trip speech synthesis and recognition, or from MEDSAGE-style prompting calibrated to the measured Whisper error profile. The non-bridge text keeps whatever background noise the transcript level implies. The target sub-query for the next hop keeps the canonical, clean entity string. Rejection sampling keeps only chains that reach the gold answer. The model is thus rewarded for inferring "Christopher Nolan" from "Christopher Nolen," or from surrounding context, before it searches.

The third component is a secondary analysis of compressors. We test whether CompAct's completeness token and Superposition Prompting's pruning discard corrupted gold evidence at higher rates than clean gold evidence.

Baselines

The naive reference point is off-the-shelf IRCoT and an untrained CoRAG-style chain model run directly on the transcribed corpus. We also report the same systems on the clean corpus as an upper bound. The gap between these two runs is the quantity every other method tries to close. We found no reported result on Lacuna for any multi-hop retrieval system on ASR-transcribed corpora, so both the noisy baseline numbers and the tolerance points are new measurements. For orientation on clean text, CoRAG reports MuSiQue retrieval recall of 47.1% against 29.0% for single-step retrieval, and the original MuSiQue paper reports 47.3 F1 for its best model against 78.0 for humans.

The strongest existing method is a CoRAG-style chain model trained on clean chains, with an LLM transcript corrector placed in front of it. The corrector follows GEC-RAG, which retrieves past recognition errors as examples for the correction prompt and reduced Persian WER from 39.09% (normalized) to 24.29% (1-best). It is the strongest competitor because it attacks the same entity errors without touching the reasoner. We run it with the same backbone family for correction and give it the same error-memory data our injector uses. Beating it means higher MuSiQue F1 and higher bridge survival at matched WER, with the margin widening on 3- and 4-hop questions.

The key ablation is uniform injection. It uses the identical training pipeline and number of chains, with the same total number of corrupted tokens spread across all words, as MEDSAGE does. It isolates whether targeting bridges matters or whether any noisy training suffices. A second ablation keeps the bridge corruption but uses the corrupted string as the target sub-query, which isolates the contribution of repair supervision.

We add two further comparisons. The first is a retrieval-side competitor that pairs the clean-trained chain model with a hybrid index, combining BM25 keyword matching with phonetic keys computed over entity spans and dense retrieval. The second is an end-to-end audio competitor, WavRAG. Its reported single-hop HotpotQA top-1 EM with GPT-4o is 0.4019, against 0.3124 for its text baseline. We run it on our spoken corpus with the same retrieval depth.

All systems share the same corpus, recognizer outputs, retrieval depth budget, and a fixed number of retrieval calls per question. All trained variants get the same number of fine-tuning chains and the same hyperparameter search budget.

Why not just…?

Why not just clean the transcripts first? MEDSAGE reports that training for robustness beat post-hoc cleaning, and that correctors hallucinate, especially small ones. Dan Liu et al. ( The USTC-NEL Speech Translation system at IWSLT 2018) found that restoration modules add their own compounding errors. A corrector working on a single paragraph also lacks the context that makes a bridge recoverable, namely the question and the previous hop. That said, a strong corrector is a real threat at low WER. The GEC-RAG-fronted baseline decides this, and we will report the WER range where it wins if it does.

Why not just augment training with ordinary noise? This is the most serious alternative. MEDSAGE-style uniform injection may already teach enough robustness. The matched-budget uniform-injection ablation answers the question directly. If it matches bridge-targeted training on bridge survival and on 4-hop F1, we will report that targeting adds nothing and recast the contribution as the benchmark and the compounding analysis.

Why not just make retrieval fuzzy with phonetic matching? When the query side is misspelled, phonetic keys may fix many hop failures cheaply, so this is also a genuine threat. Two kinds of error favor model-side repair instead: substitutions that produce real words ("caught in" for "Cotton"), which phonetic keys at the entity span do not align cleanly, and cases where the entity must be inferred from context. Results broken down by error type on the hybrid-index baseline will show where each approach wins. The two approaches are also compatible, so we report their combination as well.

Why not skip transcription and retrieve over audio? SpeechRAG reports its generator trailing low-WER cascades on long audio, and WavRAG degrades beyond two retrieved documents, while multi-hop questions need several documents joined in sequence. Running WavRAG on our 3- and 4-hop splits tests whether this limitation holds for chained retrieval.

Proposed Experiments

The first decisive check tests the compounding prediction. On the untrained baseline, we fit bridge survival $p$ per WER level and test whether the product of per-hop survival rates predicts end-to-end success across MuSiQue's 2-, 3- and 4-hop splits. We also locate the multi-hop tolerance point, in ENDOW's sense of where performance starts falling steeply, and compare it with the roughly 20% single-hop figure.

The second check is the main comparison. Bridge-targeted training should beat the corrector baseline and the uniform ablation on MuSiQue F1/EM, with the gain concentrated in higher-hop questions and explained by higher measured $p$ rather than by better final-step reading.

The third check probes transfer from synthetic to real speech, since synthesized speech misses spontaneous disfluencies (Spoken-CoQA). We have human speakers read the gold paragraphs for a few hundred MuSiQue questions, transcribe them with the same Whisper model, and require the ranking of methods to hold.

The fourth check measures how often CompAct's completeness token and Superposition Prompting's pruning drop corrupted gold paragraphs compared with clean ones. This tells us whether compression can safely be layered on top of noisy retrieval chains.