Revise Only Where the Voices Collide: Acoustically Triggered Self-Correction for Multi-Talker Transcripts

Transcribing a conversation where people talk over each other has two parts: writing down the words, and saying who spoke each word. Current systems do both in a single left-to-right pass. When two voices overlap, that pass often attributes a phrase to the wrong speaker, drops words hidden under the louder voice, or invents words from sound that leaked in from the other speaker. Large language models fix errors like these well when they are allowed to draft, critique, and revise. That loop multiplies inference time several fold, though, which is fatal for live captioning or meeting assistants. Our starting observation is that the errors cluster. Real telephone conversations contain only about 13–14% overlapped speech (CALLHOME, Fisher), yet overlap is where most attribution errors occur. A correction loop that fires only on the short stretches where acoustic evidence shows collision or uncertainty should recover most of the benefit of iterative refinement at a small fraction of the cost.

We propose a transcriber that runs one cheap streaming pass and then applies revision behind the live output, on flagged spans only. Each revision round is not the model re-reading its own text. A separate small critic, which sees both the words and acoustic confidence signals, emits a structured error report. An acoustic actor then re-decodes the flagged span with a target-speaker recognizer. That recognizer is enrolled on each speaker's own confidently attributed, non-overlapped speech from the current hypothesis. Every round therefore brings new acoustic evidence conditioned on the improved hypothesis, and the whole procedure has a latency cost that scales with the amount of overlap rather than with transcript length.

Core Claim

Iterative self-correction pays off for multi-talker transcription only when two conditions hold. First, the critique must be grounded in acoustic signals rather than in the generator's own text. Second, the revision must go back to the audio with a better query rather than paraphrasing the draft. Under these conditions, a loop that is gated by overlap and confidence and capped at a small number of rounds reduces speaker-attributed word errors in overlap regions beyond what a single-pass fused corrector achieves. It also keeps added latency bounded. A rough cost model makes the second point concrete:

$$ C_{\text{total}} \approx C_{\text{first}} + K\,\rho\,C_{\text{span}} $$

Here $C_{\text{first}}$ is the cost of the streaming first pass, $K$ is the round cap, $\rho$ is the fraction of audio flagged for revision, and $C_{\text{span}}$ is the per-second cost of one critic-plus-re-decode round. The form follows from revising flagged spans independently. Because $\rho$ tracks overlap fraction (roughly 0.15 on telephone speech and 0 to 0.4 on LibriCSS), the multiplier stays far below the two- to three-fold slowdown reported for full-output loops such as Volcano. Flagged spans are also independent, so they can be processed in parallel.

What Exists and How This Differs

Three lines of work supply the pieces.

The first line is second-pass correctors for diarization ("who spoke when"). Two papers show that a correction pass helps, and they motivate this proposal. Lexical Speaker Error Correction re-labels speaker tags with a RoBERTa encoder over roughly 30-word windows and cuts word diarization error by 15–30% relative. It sees only text and discrete speaker IDs, and its authors name soft acoustic confidence as the missing ingredient. DiaCorrect takes the opposite route: it corrects speaker-activity tracks from log-Mel features with no language model. Both run exactly one pass, both are built for two speakers, and neither can change words, only labels.

The second line is instruction-following or adapted foundation models for overlapped speech. These supply our first pass and our actor. MT-LLM feeds Whisper and WavLM embeddings into a LoRA-tuned Llama-2-7B and reaches 5.2% WER on two-speaker LibriSpeechMix. It decodes in one pass, is slow, and is weak on order-dependent instructions. Whisper with a sidecar separator and target-talker identifier reaches 4.66% WER on Libri2Mix, 16.79% on Libri3Mix, and 7.97% for target-talker ASR. It requires a clean 3-second enrollment clip and struggles with late-entering speakers. t-SOT streams tokens from all speakers on one output channel using a channel-change token and reaches 9.0% average WER on LibriCSS. Its emissions are final, so early channel mistakes cascade. We reuse t-SOT as the streaming first pass and the target-talker Whisper as the re-decoder. What we add is a way to revisit t-SOT's committed decisions.

The third line is refinement methods from text. These supply the loop design and its constraints. REFINER shows that a small fine-tuned critic emitting structured error reports (type, position, hint) can steer a frozen reasoner, improving it by 3.5–6.8 points. We adopt that critic format. ReLoc treats revision as local search guided by a learned estimate of distance-to-correct, using 37% fewer tokens than tree search. This motivates our local, span-level edits and a learned stopping signal. Meta-Reflection pushes the other way: it folds reflection into a single forward pass and cuts first-token latency from 951 ms to 246 ms. This is a direct challenge to any multi-round design, and we test it head-on as a distillation arm. The "self-critique blindness" the file notes for self-refinement constrains our design: the critic is never the generator.

Finally, the acoustic self-enrollment literature collected under iterative feedback for audio refinement re-embeds intermediate separated audio as a better speaker query, for example in iterative refined adaptation for speaker extraction. That work never uses a transcript or a lexical critic to decide what to re-extract, and it has no latency bound. Our loop is, in effect, self-enrollment steered by a language-aware critic and restricted to collision spans.

Research Program

The first pass is a streaming t-SOT transducer. Alongside its tokens it logs per-token posteriors and the frame-level overlap probability from an end-to-end neural diarizer (EEND), the model family introduced in End-to-End Neural Speaker Diarization with Permutation-Free Objectives.

A trigger flags a span when any of three conditions holds: the overlap probability exceeds a threshold, token confidence drops, or channel changes occur densely. The same trigger marks likely leakage words. Following the leakage test in SSGD, these are words whose acoustic evidence correlates more with the other channel than with their own.

The critic is a small encoder-decoder, T5-base scale as in REFINER. Its input is the flagged span plus about a minute of surrounding transcript, together with compact acoustic summaries: token confidences, overlap probability, and cosine similarity of the span's speaker embedding to each speaker's running profile. Its output is zero or more reports of the form ⟨error type, token span, hint⟩. The error types are speaker swap, missed words, leaked or hallucinated words, and boundary misplacement. The critic also outputs a stop probability, trained ReLoc-style to predict whether any further revision would help.

The actor acts on each report. For a swap or boundary error, it rebuilds enrollment for the implicated speakers from their high-confidence, non-overlapped segments in the current hypothesis. It then re-decodes the span with target-talker Whisper, once per candidate speaker. For a leaked word, it deletes the word if re-decoding under the other speaker's enrollment recovers it. The revised span replaces the old one, and the critic runs again, up to $K$ rounds (we expect $K \le 3$).

The loop runs behind the live output with a maximum delay $D$ seconds. Captions therefore appear immediately and are silently corrected within $D$. We report accuracy as a function of $D$.

The critic is trained without a proprietary teacher. Its training data combine two sources: synthetic corruptions of reference transcripts (turn swaps, deletions under overlap, inserted leak words), following Lexical SEC's curriculum, and the real error patterns of the first-pass system on LibriMix, LibriCSS-style simulations, and Fisher/CALLHOME training data. Labels come from aligning hypotheses to references.

Scope is two and three speakers. We use Libri3Mix to test the ability to go beyond two speakers, which every existing corrector lacks. CHiME-5 serves only as a stress test, given its 81.3% distant-microphone baseline WER in the CHiME-5 challenge paper.

Baselines

The naive reference point is the first pass alone: t-SOT streaming output with no correction. On LibriCSS the strongest reported number is t-SOT's 9.0% average WER (t-SOT). We retrain t-SOT on the same simulation data we use for everything else so the comparison is on identical inputs. Beating it means lower speaker-attributed WER overall, with most of the gain in the higher-overlap LibriCSS subsets, at an added delay $D$ of a few seconds and less than 1.5× total compute.

The strongest offline single-pass recognizers are target-talker Whisper (4.66% Libri2Mix WER, 16.79% Libri3Mix WER) and MT-LLM (5.2% WER on LibriSpeechMix). We run both on our test sets and measure their latency on the same GPU. They tell us whether a slow, strong single pass already matches a fast first pass plus local repair. Beating them means comparable or lower WER at a fraction of their wall-clock time.

The strongest correctors are Lexical SEC, whose best reported number is Fisher word diarization error reduced from 2.26 to 1.53 with real-data fine-tuning (Lexical SEC), and DiaCorrect, which reports a CALLHOME Part 2 DER drop from 8.62% to 7.91% (DiaCorrect). We re-implement Lexical SEC on our first-pass outputs using the same training transcripts and the same corruption budget. On Fisher and CALLHOME, beating it requires lower word diarization error than the re-implementation and, ideally, than 1.53 on Fisher.

The key ablation is a fused single-pass corrector. It is our critic with its report head replaced by a direct label-and-edit head. It sees exactly the same text and acoustic features, runs once, and never re-decodes audio. It is matched in parameters and training data. If this arm ties the full loop, the claim that iteration with fresh acoustic evidence matters is false. Further ablations remove the trigger (revise everywhere) and remove self-enrollment (re-decode with the first-pass speaker profiles).

Why not just…?

Why not just add acoustic confidence to Lexical SEC and run it once? This is the most serious threat. It is exactly the gap Lexical SEC's authors identify, and it is cheap. A single-pass corrector, however, can only re-label words that already exist. It cannot recover words masked under overlap or test a hypothesis against the audio. The fused single-pass ablation above is this alternative built as strongly as we can build it. We will report its error rate separately for swap, deletion, and insertion errors. We expect the two methods to tie on swaps and the loop to win on deletions and leak insertions. If the loop does not win on those, we will say so.

Why not just sample N transcripts and rerank? SRT and ReLoc both report that targeted revision beats best-of-N at matched compute in text and code, and ReLoc finds that best-of-N saturates quickly. Neither result concerns speech, so we run best-of-N on the same flagged spans. It samples the target-talker decoder with the critic acting as a reranker, and we match it to the loop's GPU-seconds. Because it uses the same trigger, the comparison isolates editing against resampling.

Why not just use a bigger single-pass model such as MT-LLM? MT-LLM is accurate on simulated mixtures but decodes a 7B model over long dual-encoder token sequences, which rules out streaming. It is also weakest on order-dependent attribution, which is the kind of error our loop targets. Our latency-versus-WER curves place it directly on the same axes.

Why not just distill the loop into one pass, as Meta-Reflection does? If distillation works, it is the better product, so we test it. We train a single-pass student on the loop's final outputs. On MBPP, Meta-Reflection matched iterative baselines. Our hypothesis is that distillation will fall short here because the loop's gains come from re-querying audio with enrollment derived from the hypothesis, and a static student has no access to that at inference time. The size of the gap between student and loop measures how much of the benefit is genuinely iterative.

Decisive Experiments

The benchmarks are LibriCSS for real-room, distant-microphone overlap from 0% to 40%, SparseLibriMix for a controlled overlap sweep, Libri2Mix and Libri3Mix for two versus three speakers, and Fisher and CALLHOME for real telephone speech and word diarization error. The metrics are speaker-attributed WER, word diarization error, and WER broken down by overlap bin and error type. For compute we report the flagged fraction $\rho$, rounds used, GPU-seconds per audio-second, and the revision delay $D$.

Three results would settle the claim.

First, on the SparseLibriMix sweep, total compute should rise roughly linearly with overlap ratio, as the cost model predicts. Accuracy on 0%-overlap mixtures should not degrade. The LibriMix authors report that dense-overlap-trained systems hallucinate speech during single-speaker segments, and a gated loop must not make that problem worse.

Second, the full loop must beat the fused single-pass ablation and best-of-N at matched compute on overlap-bin error, with the gap concentrated in deletions and leak insertions.

Third, on Libri3Mix the loop must improve on the 16.79% target-talker Whisper result. That would show the correction extends past the two-speaker ceiling of every existing corrector.

If the fused single-pass ablation ties the loop, or the distilled student closes the gap, then iteration is unnecessary for this problem. That would itself be a useful negative result for the self-correction literature.