Speech recognizers that look at their surroundings make fewer mistakes. A robot that sees a lamp, or a lecture system that reads the current slide, can push its transcription toward words that are actually on screen. But speakers constantly say words that the scene only implies. A lecturer standing in front of a slide about "attention heads" says "key-value cache". The term is not on the slide, but it appears two pages later in the linked paper. A user in a kitchen asks for "the charger for the tablet", and the charger is out of view.
Every visually grounded recognizer we know of matches spoken words directly against what is visible, so it cannot help with these words. The natural fix is to chain lookups ("visible thing → related document → likely word"), as multi-hop question answering does. That kind of chained retrieval usually costs seconds of repeated text generation, which is fatal for a speech interface.
We propose a small learned module that reads the frozen speech model's own internal states, performs one hop of retrieval from the visible context into a larger linked store (papers, neighboring slides, a scene knowledge base), and filters the retrieved candidate words by how they sound against the audio. The surviving words then bias ordinary beam search. The central empirical question is whether this selective hop beats the obvious alternative, which is to dump every linked word into a big bias list and let phonetic matching sort it out. We predict an answer that depends on scale: flat expansion should be competitive when the linked store is small and should degrade as it grows. We design the experiments so they can show this prediction is wrong.
Core Claim
Residual errors in environment-aware speech recognition are concentrated on "second-hop" words: words that are absent from the visible context but reachable from it through one link. Visible-context biasing cannot fix them by construction. Flattening all reachable words into one bias list fixes some of them but dilutes the bias and inserts unspoken terms as the reachable set grows.
A retrieval query built from the speech model's middle-layer states and the visible context can choose which part of the linked store to open, at millisecond cost. This follows the idea Optimizing Multi-Hop Document Retrieval Through Intermediate Representations (L-RAG) showed for text language models: a small network maps middle-layer hidden states into a retriever's embedding space so that the second-hop document is fetched without generating a textual query. L-RAG reports about 2.8s latency versus more than 20s for iterative retrieve-and-generate on 2WikiMultiHopQA.
Neither half of this combination is new on its own. The new part is where it is applied: to recognition itself rather than question answering, with phonetic filtering as the second stage. The main contribution is to measure honestly when the hop is worth its cost.
Where This Sits
Three lines of prior work motivate the gap.
The first is environment-aware recognition. Can Visual Context Improve Automatic Speech Recognition for an Embodied Agent? (RoSI) turns dense captions of the robot's camera view into a prefix tree of bias words for beam search. VHASR treats image patches as "vision hotwords" matched against audio features. AVFormer injects CLIP frame features into a frozen speech model through small adapters. M³AV uses slide OCR as a bias list and reports a 37.8% relative decrease in biasing-word error rate. All four assume the spoken word is itself visible. None of them retrieves beyond the frame.
The second line motivates why reasoning cannot simply be handed to a bigger audio model. SAKURA shows that audio-language models perceive a single fact well but fail to use it as a premise. DeSTA2 scores 65.8% on multi-hop questions from audio versus 93.3% from ground-truth text.
The third line supplies the retrieval ingredients.
- Multi-Modal Retrieval For Large Language Model Based Speech Recognition shows that retrieval keys built from the audio beat text-only keys. It reports next-token accuracy of 69–85% versus 16–34%, at a 3–4 ms overhead. Its retrieval is single-hop, and it has no visual context.
- H-PRM and Retrieval Augmented Correction of Named Entity Speech Recognition Errors show that confusions in speech are phonetic, not semantic. Phoneme-to-phoneme matching reaches 92.57% R@1 in H-PRM. We therefore do not use semantic retrieval to pick the final words. We use it only to pick the region of the store, and phonetics picks the words.
H-PRM also constrains the design. Its phonetic pre-retrieval scales from 150 to 3,800 hotwords while losing only 2 points of pre-recall (99.0% to 97.0%). This is exactly the evidence that makes flat expansion a serious competitor, and we treat it as one.
The iterative-training literature enters as a constraint. We have no gold "hop paths", so the selector must learn from its own mistakes through rounds of hard-negative mining, which is the strategy H-PRM uses. Iterated schemes are known to multiply cost by the number of rounds. Iterated learning for emergent systematicity in VQA had to abandon full parameter resets on larger data for this reason. We keep the backbone frozen, as AVFormer does, and cache its states once, so each round retrains only a module of a few million parameters.
Research Program
Benchmark. We first build the benchmark the question needs, mostly from existing data. M³AV links 1,113 lecture videos to 24,956 slides and 767 papers. For every reference biasing word, we label it hop-1 if it appears in the current slide's OCR. We label it hop-2 if it is absent from the current slide but present in the same lecture's other slides or in the linked paper. Otherwise we label it unreachable. This split costs no new annotation.
We also create a controlled scaling axis by enlarging the linked store in stages:
- current slide only;
- the whole lecture's slides;
- the lecture's slides plus the linked paper;
- all of that plus papers from other lectures in the same venue, as distractors.
Each stage increases the number of phonetically confusable candidates.
Embodied check. For a second, embodied check, we write around 200 new instructions over RoSI-style simulator scenes. Each instruction refers to an object that is occluded or out of view but related to a visible object, and each is paired with a small scene knowledge base. The instructions are recorded by native, non-native, and synthetic voices, following RoSI's protocol.
System.
- A frozen speech backbone produces encoder states over the utterance. We use both a Whisper-class model and wav2vec 2.0, so we can compare against RoSI's backbone.
- A small network reads a pooled middle-layer state together with an embedding of the visible context (slide OCR or scene caption).
- The network outputs a query in the embedding space of a frozen text retriever that indexes chunks of the linked store.
- Terms are extracted from the top retrieved chunks, compared phonetically against the audio (H-PRM style), and the survivors are added to the visible-context bias words for shallow-fusion beam search.
Before training the selector, we run an L-RAG-style probe across encoder depths. The probe tests whether speech encoders have a middle band that carries "what is being talked about" better than early acoustic or late token-refinement layers. The answer determines which layer the selector reads.
Training. Positive chunks for each training utterance are the chunks that contain a hop-2 word actually spoken, which we read off the reference transcript. Negatives are mined over rounds. After each round, we score every training utterance with the current selector and keep chunks that rank highly and contain phonetically confusable but unspoken terms. We use this phonetic proxy instead of re-running beam search, so each round costs one pass of a tiny network over cached states. We will report total training GPU-hours against a single supervised pass. We will also report whether the proxy mining matches mining from real decoding errors on a subsample.
Baselines
Naive reference points. The first is the frozen backbone with no biasing. The second is visible-context biasing: slide OCR for lectures, and RoSI's caption-derived prefix tree for scenes. For visible-context biasing on M³AV, the reported figure is the 37.8% relative biasing-word error rate decrease from M³AV. No absolute biasing-word error rate for that setting was found on Lacuna, so we will reproduce it on our backbone. On the embodied side, RoSI reports 8.48% WER versus 20.83% for plain wav2vec 2.0. By construction, these baselines should gain almost nothing on hop-2 words. Confirming that they do not is the first sanity check.
Flat expansion with phonetic pre-retrieval. This is the strongest competitor. Every term in the linked store at a given scale is placed into one list, H-PRM-style phonetic pre-retrieval selects the top candidates against the audio, and those candidates bias decoding. It uses the same backbone, the same phonetic matcher, the same bias weight tuned on the same development split, and the same number of final bias words as our system. The only difference is that no semantic hop selects the region of the store. The H-PRM figure makes this baseline credible: top-50 pre-recall drops only from 99.0% to 97.0% between 150 and 3,800 hotwords, while SeACo-Paraformer without pre-retrieval falls from 94.6% to 71.9% post-recall. Beating this baseline means lower hop-2 biasing error at matched unspoken-term insertion rate, at the largest store sizes. We also require that the advantage grow with store size.
Two-pass cascade. A recognizer produces a first transcript. A text language model reads the transcript and the visible context, retrieves from the linked store in the IRCoT/ITER-RETGEN manner (alternating generated reasoning with retrieval), and proposes terms. A second decoding pass is then biased with those terms. SAKURA's finding that text-side reasoning is much stronger than audio-side reasoning makes this a real accuracy threat. We run it with the same retriever and store, and report both accuracy and wall-clock latency. We have no reported recognition result for this cascade on Lacuna. The closest reported cost figure is more than 20s for iterative retrieval on 2WikiMultiHopQA, from L-RAG. Beating the cascade means matching its hop-2 error within a small margin at roughly a hundredth of its latency.
Ablations isolating the key ingredient. First, we replace the middle-layer speech state with the visible context alone, which is a text-only query from the slide. This tells us whether listening matters for choosing the hop. Second, we replace the middle-layer state with the final-layer state, and separately with the decoded prefix text. These test the layer hypothesis.
Why not just…?
Why not just put everything reachable into one big bias list? This is the alternative most likely to win, and the proposal is built around testing it. At small store sizes we expect it to match our system, and if it keeps matching at the largest scale, the honest conclusion is that phonetic pre-retrieval already solves second-hop biasing and the semantic hop is unnecessary. Our reason to expect otherwise is that H-PRM's 3,800-word lists are curated hotwords, while a lecture plus its paper plus distractor papers contains many more near-homophone technical terms. Phonetics alone cannot tell which "Gaussian process" is relevant. The store-size sweep settles this directly.
Why not just transcribe first and let a text language model reason about missing terms? SAKURA shows text reasoning is far stronger than audio reasoning (93.3% vs. 65.8% for DeSTA2), so this cascade may be more accurate. Two things work against it. The first transcript is least reliable exactly on the rare words that matter; Multi-Modal Retrieval For Large Language Model Based Speech Recognition measures 16–34% versus 69–85% next-token accuracy for text versus audio retrieval queries. The cascade also needs multiple generation passes. We report the cascade's accuracy and latency head to head. If it wins by a wide margin, our contribution reduces to a cheaper approximation, and we will say so.
Why not just give a large audio-language model the slide and the paper in its prompt? Long paper contexts are expensive per utterance, and SAKURA indicates that these models do not reliably connect what they hear with knowledge they have. VHASR's 333M grounded model beating 1.5B Whisper on ADE20k (9.53 vs. 14.08 WER) suggests that targeted grounding beats scale in this regime. We include one open audio-language model with the slide plus retrieved paper chunks in context, at matched retrieval, as a reference point.
Proposed Experiments
Error decomposition. The first experiment splits biasing-word errors into hop-1, hop-2, and unreachable on M³AV for the no-bias and slide-biasing baselines. The motivating claim fails if hop-2 words are a negligible share of residual errors after slide biasing.
Store-size sweep. The central experiment compares our selector with flat expansion as the linked store grows. Metrics are hop-2 biasing-word error, overall WER, the insertion rate of bias terms that were not spoken (the hallucination measure, reflecting AVFormer's "hands" for "ends" failure), and added latency per utterance. The prediction is a widening gap in favor of the selector as the store grows, with equal or lower insertions.
Layer probe and ablations. These test whether middle speech-encoder layers are better hop queries than final layers or context-only queries. If the context-only query matches the speech-state query, the listening component is decorative.
Cascade comparison. This places our system on an accuracy–latency plane against the two-pass cascade.
Training cost. The last in-domain experiment reports GPU-hours and hop-2 error across mining rounds, against a single round with random negatives. The iterative mining has to earn its extra rounds.
Embodied transfer. The embodied instruction set asks whether the same selector idea transfers from documents to a scene knowledge base with occluded referents, with RoSI as the visible-only reference.
References
- Optimizing Multi-Hop Document Retrieval Through Intermediate Representations
- Can Visual Context Improve Automatic Speech Recognition for an Embodied Agent?
- VHASR: A Multimodal Speech Recognition System With Vision Hotwords
- AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASR
- M³AV: A Multimodal, Multigenre, and Multipurpose Audio-Visual Academic Lecture Dataset
- SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Information
- Multi-Modal Retrieval For Large Language Model Based Speech Recognition
- H-PRM: A Pluggable Hotword Pre-Retrieval Module for Various Speech Recognition Systems
- Retrieval Augmented Correction of Named Entity Speech Recognition Errors
- Iterated learning for emergent systematicity in VQA