Training Speech Recognizers Against Whole Domain Lexicons

Speech recognizers that do well on clean audiobooks fall apart on specialized talk, such as clinical dictation or particle-physics lectures, because the rare technical words those settings depend on barely appear in the text the model learned from. The usual fix is contextual biasing: give the recognizer a list of domain terms and let it attend to them while decoding. This works for short lists and fails for long ones. Models trained on lists of a few hundred terms either lose accuracy or start "hearing" listed terms that nobody said. A real domain lexicon, such as a hospital's drug and procedure names or every particle and detector name in CERN documents, has tens of thousands of entries.

We propose to train the biasing module against the entire lexicon at once. The lexicon is split across GPUs, each GPU attends over its own slice, and the slices are recombined exactly using the distributed-softmax technique that large-vocabulary language models use to spread their output layer across devices. Our claim is not mainly about speed. We claim that the hallucination failure is partly a mismatch between training and deployment: models are trained with small, entity-rich lists and tested with large lists full of distractors. Training at deployment-scale list sizes, which memory limits have so far ruled out, should teach the model to tell a spoken entity from a merely listed one.

Core Claim

Contextual biasing in speech LLMs fails at scale for two separate reasons, and the literature has run them together. The first is mechanical. Putting a long list into the text prompt dilutes attention. Contextual Biasing Speech Recognition in Speech-enhanced Large Language Model reports exactly this: prompt biasing works at three entries and breaks down near 100. That paper also shows that fusing encoded bias entries through attention placed after the acoustic projection ("post-fusion") is the better design, with a 35% relative reduction in bias-word error rate on LibriSpeech. It trains and evaluates only with lists of up to about 500, and the authors flag larger lexicons as degrading or exceeding memory.

The second reason is statistical. ContextASR-Bench documents a "hallucination trap": given fine-grained entity lists, large audio-language models copy listed entities even when the audio does not contain them.

We hypothesize that the second failure follows partly from the first. Because large lists could not be trained on, training lists have a high fraction of entries that were actually spoken, and the model learns that a listed entity is probably present. The proposal has three parts:

  1. Make full-lexicon post-fusion trainable by sharding the lexicon and its attention.
  2. Add an explicit "nothing here" option that competes with every lexicon entry in one global softmax.
  3. Test whether training with realistic distractor ratios cuts false entity insertions while keeping the gain on true entities.

Mixing two words at random could look like a fix, but ordinary negative sampling of a few hundred distractors does not reproduce the deployment distribution. That is the comparison the experiments must win.

Research Program

Base system and fusion module. We start from a speech LLM with a frozen acoustic encoder and a trainable post-fusion biasing module in the style of the SpeechLLM biasing paper, which supplies the fusion design. The base must be large. Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Prompting found that language-model-based correction does not emerge below roughly 1.5B parameters. We therefore use a 7B-class model, comparable to the Qwen2.5-Omni-7B system evaluated on ContextASR-Bench.

Sharding the lexicon. A small trainable text encoder turns each lexicon entry into a key and value vector. With tens of thousands of entries, the costly part is not storing the keys. It is three things:

  • running the encoder forward and backward over every entry at each training step;
  • the frames-by-entries attention scores that must be kept for the backward pass;
  • the gradient back into the encoder.

We split the $E$ entries into $p$ equal shards, one per GPU. Each GPU encodes its shard and computes, for every compressed audio frame, attention scores against its entries.

Exact recombination. The shards are recombined with the online distributed softmax from Balancing Pipeline Parallelism with Vocabulary Parallelism. That paper shards a text model's output vocabulary across pipeline stages and normalizes the softmax with a single synchronization. We reuse its technique and change what is being sharded: a runtime-supplied, per-deployment set of entities rather than a fixed token vocabulary.

The null slot. We add a learned null slot $n$ whose score competes globally with all entries. For an audio frame query $q$, the fused bias vector is

$$ b(q) = \frac{\sum_{k=1}^{p} \sum_{j \in S_k} e^{s_j(q)} v_j}{e^{s_n(q)} + \sum_{k=1}^{p} \sum_{j \in S_k} e^{s_j(q)}}, $$

where $S_k$ is the set of entries held by GPU $k$, $s_j(q)$ is the scaled dot-product score of entry $j$, $v_j$ is its value vector, and $s_n(q)$ is the null score. Each GPU sends only its local maximum, its local sum, and its partial weighted sum. One reduction recovers $b(q)$ exactly, so sharding changes cost and not the computation. The null slot gives the model a principled way to say "no listed entity here." Because it is normalized against the whole lexicon, its calibration is learned under realistic competition.

Placement across the pipeline. Naive manual placement puts the bias encoder and fusion on the pipeline stage holding the acoustic projection. We expect this to create the same terminal-stage skew that the Vocabulary Parallelism paper measured for output layers, where one layer cost up to 5× an ordinary block. We also expect it to interact badly with the near-zero backward cost of the frozen encoder identified by Efficient Distributed MLLM Training with Cornstarch. Cornstarch partitions multimodal pipelines by measured forward and backward times but treats inputs as fixed-size. Here the lexicon size varies by deployment, from hundreds to over 50k entries.

Rather than build a general automatic partitioner like Pfeife or Merak, whose partitions are fixed at compile time, we fit a simple profiled cost model. The model chooses the shard count and stage assignment from lexicon size and batch shape. We compare it against hand placement. This is deliberately modest: the systems layer is what makes training feasible, not the scientific contribution.

Training curriculum. Training lists contain all entities spoken in the utterance plus distractors. We vary the ratio of distractors to spoken entities from about 10 (today's regime) to the full domain lexicon (about 10^4). Distractors are drawn either uniformly or with deliberately hard, phonetically similar choices.

Baselines

Naive reference. The reference point is the same speech LLM with no biasing, and separately Whisper-Large-v3, both run contextless. This shows how much any lexicon access helps.

Prompt biasing. The second baseline puts the lexicon in the text prompt, truncated to what fits, following the prompt variant of the SpeechLLM biasing paper. This is the zero-training option practitioners reach for first.

Retrieve-then-fuse (strongest competitor). The strongest competitor is the published post-fusion method trained at its normal list sizes (up to 500), with a retrieval front end at test time. The retriever takes a first-pass transcript, embeds it, and also matches it against the lexicon by phonetic edit distance, then passes the top-k entries (k ∈ {50, 100, 500}) to fusion. This is what a careful engineer would build, and it threatens the idea directly because it sidesteps the memory problem entirely. The closest reported figure is the 35% relative bias-word error reduction on the LibriSpeech Rare5k evaluation reported in Contextual Biasing Speech Recognition in Speech-enhanced Large Language Model. For the retrieval-augmented version, there is no reported result found on Lacuna. On ContextASR-Bench the reference is Qwen2.5-Omni-7B's 39.9% relative WER and 42% relative named-entity false-negative reduction over Whisper-Large-v3 on the dialogue subset (ContextASR-Bench). The file gives only relative values, so we will rerun the absolute numbers ourselves.

Domain fine-tuning. The fourth baseline is LoRA or full fine-tuning on in-domain audio where it exists. ADMEDVOICE reduces Polish clinical WER from 24.03% to 9.5% with Whisper-medium. LHCP-ASR reaches 13.6% and 15.0% WER on its two test sets with an in-domain system.

Fairness controls. All learned baselines share the same base model, the same training audio, and the same tuning budget in GPU-hours. Retrieval baselines get their k and retriever tuned on the dev split. All methods see the identical lexicon at test time.

Ablations isolating the key ingredient. Two ablations matter most:

  • Train the sharded full-lexicon model but evaluate it with retrieved top-k lists. This separates the effect of training at scale from inference at scale.
  • Train with in-batch negative sampling of 500 distractors on a single GPU. This tests whether cheap negatives already capture the benefit.

A further ablation removes the null slot.

What beating the strongest baseline means. The proposal wins if it lowers named-entity false-negative rate below retrieve-then-fuse at matched or lower false-alarm rate, on at least two of the three domain benchmarks.

Why not just…?

Why not just retrieve the top hundred entries and keep lists short? This is the most serious alternative. Retrieval needs a query, and the natural query is a first-pass transcript. Yet the "homophone trap" described in WER we are and WER we think we are means the first pass replaces rare terms with common sound-alikes, so retrieval misses precisely the entities that matter. Phonetic matching softens this but brings in sound-alike distractors. We will measure retriever recall@k of spoken entities directly. If recall@100 is close to 1 on ContextASR-Bench and LHCP-ASR, and the training-scale ablation shows no gain, then full-lexicon training is unnecessary and we will report that as the result.

Why not just sample a few hundred hard negatives and train on one GPU? This may capture much of the benefit, and the in-batch negative ablation exists to find out. Our expectation is that it falls short because the null slot's calibration depends on the full normalization: a softmax over 500 entries teaches different confidence than one over 50,000. If the cheap version matches on false alarms, the sharding contribution shrinks to an inference convenience.

Why not just run chunked, memory-efficient attention over the lexicon on one device? At inference this may well suffice, and we will say so. Chunking the attention computation removes the score-matrix memory problem but not the compute of encoding and backpropagating through tens of thousands of entries at every training step. That compute is what sharding spreads out. The systems experiments report wall-clock time per step for both options.

Why not just fine-tune on in-domain audio? It works when such audio exists, as the ADMEDVOICE results show. But a new hospital or conference has a lexicon long before it has transcribed audio, and lexicons change. We test both alone and combined. A complementary gain on ADMEDVOICE, or a large gain where no in-domain audio is used, would justify the biasing route.

Proposed Experiments

Benchmarks and lexicons. We evaluate on four benchmarks:

  • the ContextASR-Bench fine-grained tier, with each test item's entity list expanded to the full per-domain entity inventory from the 300k-entity pool;
  • LHCP-ASR, with a lexicon mined from its 1.5B-token CERN text corpus;
  • ADMEDVOICE, with a Polish clinical-term lexicon;
  • the LibriSpeech Rare5k bias evaluation, for continuity with the published post-fusion result.

Metrics. We report overall WER, NE-WER, NE-FNR, and an entity false-alarm rate: listed entities emitted but not spoken. The false-alarm rate is our direct measure of hallucination.

Decisive check. The decisive check is a curve of false-alarm rate and NE-FNR against test lexicon size (10^2 to 5×10^4), with one curve per training distractor ratio. The hypothesis predicts that models trained at small ratios show false alarms rising with test size, while models trained at full scale stay flat. If the curves overlap, the train-test mismatch account is wrong.

Systems checks. We report per-stage peak memory, the spread of memory across stages, and step time for hand-placed fusion versus cost-model placement as lexicon size grows. The target is memory spread comparable to the under-2.5 GB reported by Vocabulary Parallelism, and no out-of-memory failures at 5×10^4 entries on a 32-GPU setup.