A speech recognizer cannot score every possible word sequence. It keeps a "beam," a short list of the best partial hypotheses at each moment, and throws the rest away. In conversation, the words most likely to be thrown away are often the ones that matter most: a name, a place, or a product that the user mentioned three turns ago and is now referring back to. A general-purpose language model assigns these words low prior probability, so they fall outside the beam before the acoustic evidence can rescue them. Once a word is pruned, nothing downstream can bring it back. That includes the question-answering system, the retriever, and any reranker, and it is why text models collapse on spoken dialogue (FlowQA drops from 75.1 to 34.7 F1 on Spoken-CoQA, as reported in End-to-end Spoken Conversational Question Answering).
We propose to separate two jobs that current systems merge. The conversation model should decide which hypotheses are kept alive, not which hypothesis wins. Concretely, a cross-turn entity predictor marks hypotheses that continue toward a plausible antecedent. Those hypotheses are protected from pruning, but their scores are not boosted. The width of every beam, protected or not, is set per frame by a statistical model of how far the correct path tends to fall behind the best path. That model is fit with extreme value theory, the statistics of distribution tails, so the beam has an explicit, tunable budget for search errors instead of a hand-tuned margin. The claim to test is that protection without score boosting recovers antecedent words that static and confidence-driven beams lose. It should do so at equal or lower compute, and without the false insertions that come from pushing context into the scores.
Core Claim and Positioning
Two older lines of work each solve half the problem. Using Dialog-Level Knowledge Sources to Improve Speech Recognition (MINDS) showed that predicting the next turn's concepts from dialogue state cuts branching factor from 63.8 to 14.2 and raises semantic accuracy from 32.1% to 58.4%. It did this with hand-written goal trees that served as a hard grammar, so any off-script utterance failed. A confidence-guided dynamic pruning approach (CGD) widens the beam when frame-level acoustic confidence is low. It reaches a time factor of 0.23 with no WER loss on Verbmobil '96, but it knows nothing about discourse, and its sigmoid parameters are tuned offline. Both papers motivate this proposal. Neither offers what we need: MINDS shows that discourse shrinks the search space, and CGD shows that adaptive widths beat fixed ones. Our design keeps MINDS's insight but makes context a soft protection rather than a constraint. It keeps CGD's adaptivity but replaces the hand-tuned transfer function with a calibrated tail model.
The thresholding method comes from Surprise: Result List Truncation via Extreme Value Theory. That paper fits a Generalized Pareto Distribution (GPD) to the upper tail of non-relevant retrieval scores to set per-query cutoffs on uncalibrated scales. Our use differs in one way. We fit the GPD not to competitor scores at test time but to the training-time distribution of the gold path's margin behind the best hypothesis. The quantity being bounded is therefore the probability of a search error, not a relevance p-value. Surprise also warns that tail fits break when the tail has few or noisy points. We address this by fitting offline on large forced-alignment data and conditioning only the scale parameter on a few covariates. The broader anomaly-detection evidence in An Evaluation of Anomaly Detection and Diagnosis in Multivariate Time Series supports the bet that the thresholding function matters more than model size. There, dynamic scoring lets a simple autoencoder beat complex models.
Contextual fragmentation enters as a constraint on how the entity predictor is built. Antecedents sit in earlier turns, which were transcribed and segmented separately. The predictor must therefore read across turn boundaries without letting old topics drown the current one. We reuse two techniques. Following Late Chunking, the whole dialogue history is encoded jointly before it is pooled per turn. Following Learning Contextual Retrieval for Robust Conversational Search, representations are pooled from the current turn's tokens, a design shown there to reduce historical interference under topic switches. Neither paper handles errorful ASR input or interfaces with a decoder. Here they serve as the source of candidates that the decoder protects.
Research Program
The system has three parts. First, an entity predictor runs once per turn over the full prior transcript, using the top ASR hypothesis plus lattice alternatives for low-confidence words. It outputs a small set of candidate antecedent words and phrases with probabilities. It is trained on conversational QA data where gold rewrites expose which history entities the current turn depends on. Rewrite supervision is used only as a training signal, never at run time, so we avoid the 1,100 ms autoregressive rewriting latency reported in Learning Contextual Retrieval for Robust Conversational Search.
Second, the candidates are compiled into marks on the decoder's lexical prefix tree, the tree that shares common word beginnings (Improvements in search algorithm for large vocabulary continuous speech recognition). A hypothesis is "protected" while its current word prefix is consistent with some candidate.
Third, the beam threshold is set as follows. Let $m_t$ be the margin by which the gold path trails the best active hypothesis at frame $t$, collected from forced alignments on training data. Above a base level $u$, exceedances of $m_t$ are modeled as a GPD with shape $\xi$ and a scale $\sigma(x_t)$ that depends on covariates $x_t$. The covariates are CGD's normalized acoustic confidence and whether the frame lies inside a protected prefix. The beam at frame $t$ is the $(1-\varepsilon)$ quantile of this conditional distribution:
$$B(t) = u + \frac{\sigma(x_t)}{\xi}\left[\left(\frac{\varepsilon}{\zeta}\right)^{-\xi} - 1\right],$$
where $\zeta$ is the fraction of training frames with $m_t > u$ and $\varepsilon$ is the per-frame search-error budget. This is the standard peaks-over-threshold quantile. Protected hypotheses use their own fitted scale, which we expect to be larger because the generic language model underrates entities. They therefore survive at larger margins. Final ranking is left to the unmodified acoustic and language model scores.
The analysis component asks one question: does the realized search-error rate track $\varepsilon$ on held-out data? This is the calibration property a hand-tuned sigmoid cannot offer. The decoder is a weighted finite-state transducer decoder with a neural language model, in the style of An Asynchronous WFST-Based Decoder For Automatic Speech Recognition.
Baselines
The naive reference is a static beam whose single width is tuned on development data. It is reported at several widths to trace a WER-versus-compute curve. The file reports that classical beam pruning reaches only about a 5x speedup at roughly 1% WER increase on Verbmobil (CGD). Any proposed operating point must lie strictly below this curve.
The strongest existing pruning method is CGD, reimplemented in the same decoder, with its $\alpha, \beta$ tuned on the same development set and the same tuning budget as our $\varepsilon$ and covariate model. Its reported result is a time factor of 0.23 at 0% WER degradation and 0.07 at about 1% WER increase on Verbmobil '96. We first try to reproduce these figures, then beat them. Beating CGD means equal overall WER at lower real-time factor, plus a significant reduction in antecedent-word error rate at matched compute.
The strongest competitor on the conversational question is shallow fusion: a dialogue-conditioned language model, fed the same entity candidates, is added to the decoder score, with a static or CGD beam. It receives the same predictor, the same data, and a grid search over its fusion weight. Beating it means comparable antecedent-word recovery with fewer entity insertions on turns that do not refer back.
For downstream evaluation, the strongest reported system on Spoken-CoQA is DDNet with an ALBERT-base backbone at F1 = 61.3 (End-to-end Spoken Conversational Question Answering). We feed DDNet our transcripts in place of its CMU Sphinx transcripts and keep the reader fixed. Gains over 61.3 then come only from recognition.
Two ablations isolate the key ingredients. The first uses the EVT beam without the protection covariate, which measures calibration alone. The second protects hypotheses with a fixed extra margin and no tail model, which measures protection alone.
Why not just…?
Why not just fuse a dialogue-aware language model into the scores and keep a normal beam? This is the real threat. Fusion also keeps entity hypotheses alive, because raising their scores keeps them within the beam. The difference is that fusion changes who wins: a boosted entity can overwrite acoustically better words on turns that do not refer back. Protection changes only who survives. MINDS's failure on off-script utterances is the extreme version of letting context dictate the output (Using Dialog-Level Knowledge Sources). The 2×2 experiment below (fusion on or off, protection on or off) settles it. If fusion at its best weight matches protection on both antecedent recovery and insertion rate, the protection mechanism is unnecessary, and we would report the EVT beam plus fusion as the contribution.
Why not just use CGD's acoustic confidence, which already widens the beam on hard frames? Entity words are often acoustically clear but linguistically improbable. Low language-model prior is not the same as low acoustic confidence, so CGD has no signal that tells it to widen there (CGD). The EVT-without-protection ablation, which includes the CGD confidence covariate, shows how much of the antecedent loss acoustic confidence alone fixes.
Why not decode with a wide beam and fix references afterward with N-best rescoring or a robust reader like DDNet? Rescoring cannot recover words that were pruned before the list was formed. Readers can only partly compensate: DDNet's 61.3 F1 still trails clean-text FlowQA's 75.1. Uniformly wide beams cost compute everywhere, the waste CGD documents. We include a "very wide beam plus context rescoring" system to bound what post-hoc correction achieves, and we compare at matched real-time factor.
Proposed Experiments
The first experiment tests calibration and compute on Verbmobil '96, which is real spontaneous dialogue speech. We sweep $\varepsilon$ and plot the realized search-error rate, measured by checking whether the gold path outscores the output after a near-exhaustive rerun, against the target. We also plot WER against real-time factor and average active hypotheses, alongside the static and CGD curves.
The second experiment tests reference recovery. We redecode the Spoken-CoQA audio and a text-to-speech-synthesized version of TopiOCQA that we construct. On these we report overall WER, antecedent-word error rate (error on reference words naming an entity introduced in a prior turn), and entity insertion rate on non-referring turns, all in the 2×2 fusion/protection design. TopiOCQA's topic switches test whether the cross-turn predictor wrongly protects stale entities.
The third experiment is downstream. We report Spoken-CoQA F1 with DDNet fixed against its reported 61.3. We also report retrieval MRR on spoken TopiOCQA with ContextualRetriever. For spoken TopiOCQA we found no reported result on Lacuna; the text-only 42.2% MRR serves as a ceiling, not a baseline.
Two caveats limit how far the results generalize. Both QA corpora use synthetic speech, which lacks disfluencies, and the disfluency-heavy regime is where search graphs grow most (Disfluency-aware Language Models). For that reason the compute and calibration claims rest on Verbmobil.
The decisive check is the 2×2 table. The proposal stands if protection without fusion lowers antecedent-word error at matched compute while keeping insertion rate at the static-beam level, and if the EVT beam's realized error rate tracks $\varepsilon$ within a small factor across corpora. It fails if fusion alone matches it, or if the tail fit drifts badly out of calibration on spoken dialogue.
References
- End-to-end Spoken Conversational Question Answering: Task, Dataset and Model
- Using Dialog-Level Knowledge Sources to Improve Speech Recognition
- A confidence-guided dynamic pruning approach - utilization of confidence measurement in speech recognition
- Surprise: Result List Truncation via Extreme Value Theory
- An Evaluation of Anomaly Detection and Diagnosis in Multivariate Time Series
- Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models
- Learning Contextual Retrieval for Robust Conversational Search
- Improvements in search algorithm for large vocabulary continuous speech recognition
- An Asynchronous WFST-Based Decoder for Automatic Speech Recognition
- Improving Automatic Speech Recognition for Children's Reading Assessment with Disfluency-aware Language Models