Filling Uncertain Speech-Recognition Slots from Noise-Matched Neighbors

Speech recognizers weigh thousands of competing word sequences but usually hand downstream systems only the single most likely one, the "1-best" transcript. When that transcript is wrong, the correct word is often sitting just below it in the recognizer's internal candidate list. Sometimes the beam search, which keeps only the most promising partial hypotheses, has already pruned it away entirely. This proposal treats each uncertain word position as a missing value in a partially observed record. The confident words around it are the observed attributes. The uncertain word is filled in the way k-nearest-neighbor (k-NN) imputation fills a missing table cell: find past utterances whose observed parts look most similar, and borrow what they contained at that position.

The key design choice is where those past utterances come from. We do not index clean text. We index what the recognizer itself produced on training audio, paired with the human reference transcript. A neighbor is therefore a past case where the recognizer showed a similar confusion pattern, and its stored value is the word that was actually said. We also borrow a rule from the imputation literature: fill a value only when the neighbors are close and agree, and otherwise leave the recognizer's own distribution alone. The claim is that this memory corrects the specific errors a deployed recognizer makes in a target domain. It can also restore words the recognizer never proposed. And it will not make things worse in very noisy conditions, which is where current multi-hypothesis methods break down.

Core Claim

The recognizer output we work with is a word confusion network: a sequence of time slots, each holding competing words with posterior probabilities that sum to one, plus an empty "deletion" option. We mark a slot as missing when its top posterior falls below a threshold $\tau$.

To fill a missing slot, we build a query vector from the rest of the confusion network with that slot masked out. The encoder reads context on both sides and weights each word by its posterior. We then retrieve the $k$ nearest entries from a datastore. Each entry has a key, which is a masked-slot encoding computed from recognizer output on training audio, and a value, which is the reference word aligned to that slot. The value can be the empty token. The retrieved values define a donor distribution $p_{nn}(w)$, with each neighbor weighted by $\exp(-d_i)$, where $d_i$ is its distance to the query. The final slot distribution is

$$p(w) = \big(1-\lambda\big)\, p_{asr}(w) + \lambda\, p_{nn}(w), \qquad \lambda = g(d_1, a)\ \text{if } d_1<\delta \text{ and } a>\alpha,\ \text{else } 0,$$

where $p_{asr}$ is the recognizer's slot posterior, $d_1$ is the distance to the nearest donor, $a$ is the fraction of donors agreeing on their top value, and $g$ is a bucketed lookup table fit on held-out data.

The two thresholds, $\delta$ on distance and $\alpha$ on agreement, implement abstention. When they fail, the slot keeps the recognizer's answer. Because $p_{nn}$ is not restricted to words already in the slot, the method can put probability on words that were pruned.

Three papers supply the machinery, and each is changed here in a specific way.

  • Distance-conditioned mixing comes from You can't pick your neighbors, or can you?. That paper shows that a fixed mixing weight between a language model and its retrieved neighbors is suboptimal, and that making the weight depend on nearest-neighbor distance lowers WikiText-103 perplexity to 15.50 from 16.12. Its queries, however, come from clean written prefixes.
  • The fast-retrieval tricks come from Efficient Nearest Neighbor Language Models: a small classifier that decides when retrieval is worth running, merging of redundant datastore entries, and dimension reduction of stored vectors.
  • The abstention rule comes from Parimputation, which argues that isolated records lacking close donors should be deliberately left unimputed. It shows that density-aware neighbors beat fixed-k imputation in every trial on Abalone at 20% missingness.

Our changes relative to these three are:

  • The queries and keys come from noisy, two-sided recognizer output rather than clean text.
  • The unit being filled is a time-aligned slot that may be empty (a deletion) or may contain a word the recognizer never proposed.

Research Program

Datastore construction. For each training utterance in a target domain, we decode the audio with a fixed recognizer and build its confusion network. We then align the slots to the reference transcript. The alignment yields an empty value where the recognizer inserted a spurious word, and a flagged "gap" value where it dropped a word.

Key encoder. The keys come from a small encoder trained once to predict the reference word at a masked slot. It follows the posterior-weighted-sum design of the confusion network encoder in Modeling ASR Ambiguity for Neural Dialogue State Tracking. That work compresses each slot into one vector to feed a dialogue state tracker; we instead use such vectors as search keys. After training, the encoder is frozen. Moving to a new domain then means swapping the datastore, not retraining.

Interfaces to downstream systems. The corrected confusion network can be consumed three ways:

Analyses. Two analyses matter beyond headline accuracy.

The first is off-lattice recovery. We will measure the share of reference words that were absent from the original confusion network but receive the top score after imputation. We will also measure the rate at which correct in-lattice words are overwritten.

The second targets the high-noise failure mode. Everson et al. report that when word error rate (WER) exceeds 70%, confusion networks fill up with wrong candidates and the plain 1-best becomes the safer input. Tabular imputation, by contrast, keeps its advantage at high missingness, as seen in Missing Value Imputation based on Locally Linear Reconstruction, where accuracy improved by about 34.7% over discarding incomplete records at 50% missingness. We predict that the abstention gate turns the high-noise collapse into a graceful fall back to 1-best. Testing this tells us whether the collapse comes from feeding a model the flattened confusion network indiscriminately, or is inherent to multi-hypothesis input.

Baselines

Do nothing: the 1-best transcript. This is the reference point the method must never fall below. It is especially important in the noise sweep. On SLURP, 1-best recognizer output drops the best pipeline's SLU-F1 (a slot-filling score combining word- and character-level F1) to 70.84, from 78.19 on perfect text (SLURP). Beating the baseline means a statistically significant gain at matched latency, plus no significant loss in any noise bucket.

Full confusion network with no imputation. This is Pal et al.'s encoder inside the GLAD tracker on DSTC-2, which reports roughly 71.2% joint-goal accuracy against roughly 69.4% for the top-10 list. We rerun it with the same recognizer outputs and the same training data. Our corrected confusion network is fed into the identical tracker, so the only variable is imputation. Beating it means clearing 71.2% on the reproduced setup, with gains concentrated on turns whose correct slot value was off-lattice.

Strongest parametric corrector: N-best T5. N-best T5 fine-tunes T5 on concatenated N-best lists and restricts generation to the lattice. It reports up to a 12% WER reduction over the baseline recognizer on LibriSpeech with lattice-constrained decoding, and 7.2% with a 10-best input. It is the strongest competitor because it also uses in-domain supervision to fix recognizer errors. We will fine-tune it on exactly the utterance pairs that populate our datastore, with an equal hyperparameter search budget. Beating it means lower WER on the target domain, or equal WER with clearly better recovery of rare domain words.

Retrieval over clean text: kNN-LM rescoring. This is a nearest-neighbor language model in the style of Drozdov et al., with keys from reference transcripts, used to rescore the lattice. We test the same model two ways, which isolates each ingredient:

  • Clean-key ablation. Identical pipeline, but keys encode reference text rather than recognizer output. This isolates the noise-matched keys.
  • Always-impute ablation. Mixing weight fixed at its tuned constant, with no abstention. This isolates the gate.

Why not just…?

Why not just fine-tune an error corrector like N-best T5 on the domain data? This is a serious threat. Parametric correctors absorb frequent error patterns well, and Ma et al. show real gains. The case for memory rests on rare, domain-specific words. He et al. find that non-parametric datastores help most with domain shift (Law-MT perplexity 12.64 to 12.29 with no retraining). Drozdov et al. find retrieval's benefit concentrated on proper nouns and rare adjectives. If our method does not beat equally trained N-best T5 on rare-word recall, or on adapting to a new domain by datastore swap alone, then the proposal reduces to "fine-tune T5." We will report that honestly.

Why not just add a nearest-neighbor language model trained on domain text? Its keys are clean text, but at test time the query comes from corrupted output, so the lookup is mismatched exactly where it is needed. The clean-key ablation tests this directly. If clean keys match noise-matched keys, the noise-matching idea fails. We would then fall back to the simpler kNN-LM.

Why not just retrieve similar training utterances and put them in an LLM prompt next to the confusion network? Everson et al. show large models can read delimited confusion networks. They also show the format is brittle to separators and cannot express deletions, and that small models degrade below 1-best. We include retrieval-augmented GPT-class prompting as an extra baseline on SLURP and ATIS. Our method is explicitly slot-level, can abstain, and runs at a fraction of the latency; we report both accuracy and cost. If prompting wins on accuracy, the contribution narrows to the low-latency, small-model setting.

Proposed Experiments

Benchmarks. DSTC-2 is the primary benchmark because it ships with confusion networks and has a published confusion network baseline. SLURP tests spoken language understanding under distant-microphone noise; we generate confusion networks with our own recognizer. ATIS is the in-domain case with low WER. On NMSQA and ATIS, we score how much of the gap to the N-best oracle each method closes, mirroring Everson et al., who report 81% and 33%. LibriSpeech gives a WER comparison against N-best T5.

Metrics. We report:

  • downstream task scores: joint-goal accuracy, SLU-F1, intent accuracy, exact match;
  • WER, split into words that were in the lattice and words that were off it;
  • the overwrite rate of correct words;
  • added latency per utterance, targeting the real-time budget emphasized by TOLSTOI. That paper shows shallow fusion raises the real-time factor from 0.33 to 0.85, and it rejects external lookups for this reason.

Decisive checks. Three results decide whether the idea holds:

  1. Noise-matched keys must beat clean-text keys. Without this, the core idea is falsified.
  2. The gate must keep performance at or above 1-best when noise is added to SLURP audio. We add noise to push WER past 70%, while the always-impute variant falls below 1-best as Everson et al. observed.
  3. Swapping datastores must transfer. A datastore swap from DSTC-2 to SLURP domains should recover a meaningful share of the in-domain fine-tuning gain of N-best T5, with no gradient updates.