Spending the Second Pass Only Where the Medicine Is

Speech recognizers often run in two passes. A fast first pass writes a draft transcript, and a heavier second pass re-reads the audio together with the draft and corrects it. This "deliberation" reliably lowers word error rate, but it roughly doubles the model's size and compute, and it spends that compute evenly across every second of audio. In a 15-minute doctor–patient consultation, most of those seconds are greetings, small talk, and filler. The words that determine whether a clinical note is safe are few: drug names, doses, negations such as "no chest pain," and symptom onsets. The system should spend almost nothing on the rest.

We propose a selective second pass for clinical conversations. It revisits only the short stretches of audio where the draft contains a clinical concept the first pass was unsure about. It re-reads them with a small, low-rank temporal network, and it stops refining each stretch once the clinical meaning stops changing. The claim is that clinical errors cluster in a small fraction of the audio. If that holds, targeted refinement should recover most of what full deliberation gains on clinical-concept accuracy, at a small fraction of its cost, and with fewer meaning-flipping "over-corrections."

Core Claim

Second-pass correction in speech recognition has been measured and budgeted in words, but clinical use cares about concepts. Two lines of work refine whole utterances and optimize word error rate. Deliberation Model Based Two-Pass End-to-End Speech Recognition pairs a streaming first pass with an LSTM second pass that attends to both the draft and the acoustics. Transformer Based Deliberation for Two-Pass Speech Recognition replaces those LSTMs with Transformer layers and cuts second-pass compute from 7.7 to 4.8 GFLOPS. These papers supply the method we build on. They also supply a constraint: the 2020 paper reports that deliberation can over-correct ambiguous tokens. In clinical speech, that failure mode becomes a flipped negation.

Clinical speech benchmarks motivate the shift to concepts, but they do not test refinement. PriMock57 scores off-the-shelf single-pass recognizers on Extracted Clinical Concepts Accuracy (ECCA), the share of clinical concepts in the reference that survive transcription. It finds that general engines beat medical-specific ones. Speech-to-SOAP scores a 3-billion-parameter audio-language model on Concept-F1 and reports degradation on dialogues longer than 21 minutes. Neither paper asks where in the audio the concept errors sit, or what it costs to fix them.

Our proposal differs from the deliberation papers in three ways. It decides where to refine using clinical-concept uncertainty. It makes the refiner cheap with a low-rank temporal bottleneck and additive updates. It decides when to stop per region using the stability of clinical meaning, not a fixed number of passes.

Research Program

The system has three parts.

First pass. A standard streaming recognizer emits an n-best list (its top few alternative transcripts) with per-word confidences.

Selector. A lightweight clinical tagger runs over the n-best list and marks candidate concepts: medications, doses, symptoms, anatomical sites, and the polarity of each finding (present, absent, or uncertain, following the RadGraph schema). A region is flagged when the n-best alternatives disagree on a concept's identity or polarity, or when the concept's words have low confidence. Each flagged region is padded with a fixed acoustic margin.

Refiner. The refiner operates only on flagged regions. It is a factorized time-delay network in the style of TDNN-F, in which each weight matrix is split into two thin matrices through a narrow middle dimension. Lei et al. and ASAPP-ASR show that this split keeps a wide view of time while cutting parameters. We add the semi-orthogonal constraint from Wang et al. to keep the narrow layers stable.

The refiner does not re-encode the region. Instead, borrowing from iRe2f, it computes an additive correction to the first-pass encoder states, conditioned on the competing hypotheses. iRe2f used such shortcut updates for text parsing; it cut refiner parameters by 90% and kept throughput at 152.5 versus 98 sentences per second for full re-encoding. It has never been tried on acoustic time series.

Stopping rule. The refiner may iterate up to K times per region. A region stops as soon as the tagged concept and its polarity are unchanged between consecutive iterations. This rule is chosen deliberately. CRAFT found that a fixed acceptance threshold forced extra refinement rounds that injected new clinical errors. Deliberation in translation (Xia et al.) and the other studies in the file show that gains plateau after one or two passes.

Training. We train on the transcribed subsets of the 1,653-hour unified medical dialogue corpus assembled in Speech-to-SOAP, with all PriMock57 recordings held out. The loss combines expected word error with a term that heavily weights errors on concept tokens and polarity cues.

Avoiding metric gaming. The tagger used for selection is deliberately different from the concept extractor used to score ECCA. Otherwise the system could learn to satisfy one extractor rather than to get the medicine right. We also report the agreement between the two extractors.

Baselines

Naive reference. The do-nothing reference point is the first-pass recognizer alone, with no second pass. It sets the floor for both clinical accuracy and compute.

Strongest existing method. The strongest competitor is full-utterance Transformer deliberation as described in Transformer Based Deliberation for Two-Pass Speech Recognition. It is the most compute-efficient published deliberation design, and it refines every frame, so it is the natural upper bound on what a second pass can recover. That paper reports 5.1% word error rate on Google Voice Search at 4.8 GFLOPS of second-pass compute.

That benchmark is not public, so we will retrain this architecture on our training data and first-pass outputs. It will use the same training data, the same first-pass n-best lists, and the same hyperparameter-search budget in GPU-hours as our system. For PriMock57 ECCA with any deliberation system, we found no reported result on Lacuna; for single-pass engines, PriMock57 reports rankings but the file quotes no ECCA value. We will therefore report all numbers ourselves.

Commercial reference. We will also include the best general commercial engine identified by PriMock57 as an external reference point.

What counts as beating full deliberation. Our system beats full deliberation if it matches its ECCA within the confidence interval, or exceeds it, while using at most a quarter of its second-pass GFLOPs per audio minute. It should also show fewer polarity flips, meaning concepts whose present/absent status changes from correct to incorrect between passes.

Ablations. Three ablations isolate the key ingredient:

  • selecting regions by word confidence alone, without concept awareness, at a matched compute budget;
  • replacing the low-rank refiner with a full-rank one of the same depth;
  • replacing the stability-based stopping rule with a fixed one, two, or three passes.

Why not just…?

Why not just correct the draft transcript with a text-only language model? This is the most serious threat. It is simple, and modern language models know medical vocabulary well. However, a text-only corrector has no access to the audio. Its priors push toward common phrasings, which is exactly the over-correction failure that Hu et al. 2020 describe. A negation the recognizer half-heard cannot be recovered from text. CRAFT also found that edit-conditioned text refinement underperformed full regeneration, which is costly on 20-minute dialogues. We will run a fine-tuned text corrector on the same flagged regions and report ECCA, polarity flips, and compute. If it matches our refiner, the acoustic component is unnecessary and we will say so.

Why not just train the first pass with clinically weighted word error and skip the second pass? This may capture much of the gain at zero inference cost. Our concept-weighted loss can be applied to the first pass alone, and that variant is a baseline. The open question is whether a streaming model without future context can resolve ambiguities that only right-hand context disambiguates, such as dosage units or "denies…" constructions. The gap between this baseline and our system measures exactly that.

Why not just select regions by recognizer confidence? Confidence is cheap and needs no clinical tagger. But low-confidence words are mostly disfluencies and names in small talk, while some confidently wrong words are drugs. The matched-compute confidence-only ablation tests whether concept awareness earns its place.

Proposed Experiments

Decisive test. The main test is a cost–accuracy curve on held-out PriMock57. The x-axis is second-pass GFLOPs per audio minute and CPU latency. The y-axis is ECCA, alongside word error rate. We trace the curve for full deliberation, our selective refiner, and each ablation by varying the flagging threshold and K. The hypothesis fails if selective refinement needs to flag more than about half the audio to reach full deliberation's ECCA. That outcome would mean concept errors are not concentrated.

Error localization. Before building anything, we measure what fraction of reference concept errors from the first pass fall inside flagged regions, and how much audio those regions cover. This cheap analysis can kill the idea early.

Clinical safety. Clinicians will audit a sample of polarity changes to confirm that the automated flip counts reflect real errors.

Long encounters. Following the 21-minute degradation reported in Speech-to-SOAP, we test whether cost stays roughly constant per concept as encounter length grows. The selective refiner's compute should track the number of concepts, not the length of the audio.

Stopping behavior. We plot the distribution of passes used per region. The diminishing-returns literature predicts that most regions stop after one iteration. Regions that run to K are inspected as candidates for over-correction.