Regenerate Only What Is Broken: Detector-Gated, Critic-Checked Speech Restoration

Generative speech restoration rebuilds a recording from a learned model of clean speech instead of filtering noise out of it. The most capable systems do this everywhere at once. MaskSR is a masked language model, meaning a network that fills in hidden tokens from context, run over the discrete tokens of a neural audio codec (a compressor that turns audio into a grid of integer codes). At inference it starts from a fully hidden token grid and resynthesizes every frame. Diffiner runs about 200 diffusion steps over the whole spectrogram. The results sound excellent, but the costs are documented. Fidelity to the original waveform (SI-SDR) drops because even clean stretches are reinvented. Compute is spent on frames that were never damaged. When the input carries little information, the prior can invent phonemes or drift the speaker's voice. Text error correction faced the same problem years ago and solved it by splitting the job. A cheap detector marks the broken spans, and only those spans are rewritten. That roughly tripled decoding efficiency in grammatical error correction (Chen et al.).

This note proposes the same split for speech, plus one more step drawn from multi-agent systems. A separately trained detector chooses which codec tokens to hide and regenerate. All other tokens are copied from the encoded input. A critic with different weights from the generator then checks each regenerated span for content and speaker drift. It sends back a typed verdict that selects a repair: resample, widen the context, raise guidance, or abstain and fall back to a conservative estimate. The claim is not that agents improve speech. The narrower claim is that localization and independent verification, the two ingredients that make modular text correction work, also fix two specific failures of global generative restoration: lost waveform fidelity on undamaged speech, and unverified hallucination in damaged speech.

Core Claim

The proposal makes two claims. On recordings where damage is localized in time, a detector-gated masked token model should keep most of the perceptual quality of full regeneration while recovering most of the SI-SDR lost on undamaged frames, at a fraction of the token-generation work. Damage is localized when it comes from clipping, packet dropouts, transient or intermittent noise, or intermittent reverberant tails.

Across all regimes, an external critic should cut phonetic hallucination in regenerated spans more than the generator's own confidence can. Hallucination is measured as word-level changes against the clean reference transcript. If the critic's gain disappears when the generator scores its own samples, the critic adds nothing and the claim fails.

We expect no efficiency gain under stationary broadband noise. In that regime nearly every token is corrupted, and the method should honestly reduce to global regeneration. A decisive check is that it must not do worse there.

Prior Work and Its Role

The method comes from modular text correction. Chen et al. tag erroneous spans with a fast encoder and decode replacements only for marked spans. That cut decoding steps from 21,065 to 7,647 and raised BEA-2019 $F_{0.5}$ from 65.5 to 66.9. We transfer the span-only decoding idea to codec-token grids.

The AIP-Tohoku system adds a sentence-level gate that lets clean sentences bypass correction, gaining over 3 precision points. It also shows the cost: overly conservative gates cap recall. That warning shapes our design. The detector is tuned for recall, and the critic, rather than the gate, is responsible for precision.

UCorrect uses a detector, a phonetically constrained generator and a selector on ASR transcripts, cutting false alarms from 61.6% to 41.6%. It works only on text and leaves the audio itself untouched. We move the detector-generator-selector pattern into the acoustic token space.

The generative speech work motivates the proposal by documenting the gap. MaskSR and Diffiner both report SI-SDR decreases under generative refinement. MaskSR also notes that hallucination is controlled only by a fixed guidance weight. GeCo cascades a discriminative separator into a generative corrector applied over the whole signal. It is the closest existing work, and it constrains our design: its corrector cannot recover what the first stage destroyed. Our detector must therefore read the raw input, not only an upstream estimate.

The multi-agent work supplies the verification principle, not the large-language-model machinery. Self-collaboration code generation gains roughly 30% Pass@1 by giving testing to a separate role. Re-Ex shows that stating what is wrong before revising raises correction accuracy from 25.4% to 52.2%.

We deliberately do not build a language-model scheduler of the kind used in MAIR or Audio-Agent. MAIR still needs tens of seconds per image. Self-collaboration reports that role adherence emerges only in large models. Re-Ex finds that decomposition hurts on short inputs, and per-span speech decisions are about as short as inputs get. Our "agents" are therefore small specialized networks with a fixed typed message protocol.

Research Program

The generator is a MaskSR-style masked token model over a frozen Descript Audio Codec at 44.1 kHz with 9 codebooks. It is trained on DNS 2022 clean speech with synthetic degradations, as in MaskSR. We change one thing in training. Instead of always masking from scratch, the model learns to infill masked spans given unmasked tokens that come from encoding the degraded input, so it learns to stitch coherently at span boundaries.

The detector is a lightweight bidirectional network over the input spectrogram that predicts, per frame and per codebook level, whether a token needs regeneration. Training labels are cheap and exact, because degradations are synthetic. A token is labeled "broken" if the codec code of the degraded audio differs from that of the clean audio at that position, smoothed over short windows.

The first deliverable is an empirical map of how sparse these labels actually are across degradation types and SNRs, and across coarse versus fine codebooks. This map decides where the method can help at all. If corruption is dense in every regime that matters, the localization claim fails early and cheaply.

The critic sees only the regenerated spans and their context. It combines three signals. The first is disagreement among several independent samples of the same span, scored with a frozen phoneme recognizer. The second is agreement with the input's phoneme posteriors on frames where those posteriors are confident. The third is a speaker-embedding distance to the unmasked context. It emits a typed verdict per span: accept, content-unstable, speaker-drift, or boundary-discontinuity. Each type maps to one repair action. Content-unstable widens the context window and resamples. Speaker-drift raises guidance. Boundary-discontinuity extends the mask by a few frames.

The number of repair rounds is capped. A span still flagged after the cap falls back to a predictive enhancer's output, so the system abstains rather than invents. This typed verdict is the speech analogue of Re-Ex's explanation step. An ablation that collapses it to a single accept/reject bit tests whether the typing matters.

Baselines

The naive reference points are the unprocessed input and a predictive enhancer, Conv-TasNet or NSNet2. They anchor SI-SDR and runtime. MaskSR reports NSNet2 at 2.50 MOS on DNS 2020, and EARS reports the noisy input at POLQA 1.71 and 33% WER.

The strongest competitor is global masked-token regeneration: the same generator weights, the same step budget, and a fully masked grid. This is MaskSR's own inference. It is the right opponent because it isolates gating, and the published reference point is MaskSR-M's MOS of 4.36 on DNS 2020. We found no reported SI-SDR for MaskSR on Lacuna, so we measure it ourselves.

On VoiceBank-DEMAND, Diffiner+ applied to a Wiener filter reaches NISQA 4.62, up from 3.54. On EARS, SGMSE+ reaches POLQA 3.40 and 16% WER. We run Diffiner+ and SGMSE+ with public settings as external generative references. We do not claim to outscore them unless we do.

All internal variants share generator weights, training data and tuning budget. Compute is reported as token-generation forward passes and wall-clock time on one GPU. Beating the global baseline means matching its NISQA and MOS within confidence intervals while improving SI-SDR and WER, at lower compute, on the localized-damage sets.

The ablations isolate each ingredient. Oracle gating with true labels gives an upper bound on what detection can deliver. Gating without the critic shows what localization alone buys. A self-critic scores spans with the generator's own token confidence, as in standard confidence-based remasking. The untyped critic tests whether the verdict types matter. The no-fallback variant measures how much of the hallucination reduction comes simply from abstaining.

Why not just…?

Why not just blend the input and the global regeneration with a soft mask? One can run full regeneration and then crossfade back to the input wherever an estimated local SNR is high. This is the most serious threat to the fidelity half of the claim, and it may recover SI-SDR nearly as well. We include it as a baseline, with both oracle-SNR and estimated-SNR masks.

The expected difference has two parts. Waveform blending creates phase and timbre seams that the generator never sees, so we report a boundary-discontinuity metric and listening scores. Blending also still pays for full regeneration. If blending matches gating on SI-SDR and perceived quality, we concede that half and the contribution reduces to the critic.

Why not just let the generator remask its own low-confidence tokens? Masked-token samplers already do this. Whether an outside judge beats self-assessment is the central question of the second claim. Self-collaboration and Re-Ex suggest the generator's confidence is biased toward its own outputs, but that evidence comes from text. The self-critic ablation settles it directly. If self-confidence remasking matches the external critic on hallucination-induced word errors, we drop the external critic.

Why not just use a language-model scheduler to diagnose degradations and call tools, as MAIR does for images? Speech degradations at the span level are short, overlapping and fast. Re-Ex found decomposition hurts short-form inputs, and MAIR's per-image latency is tens of seconds even after its efficiency gains. We expect a small typed critic to capture the useful part, which is independent verification, without that cost. We do not run a full language-model agent, so this remains an argued choice rather than a tested one.

Why not just use a predictive model, which is far faster? EARS reports Conv-TasNet running about 170 times faster than SGMSE+, but listeners preferred the generative model, and MaskSR's MOS gap over NSNet2 is large. The predictive model stays in our system as the abstention fallback and as a baseline row.

Proposed Experiments

The main test sets are built from DNS 2020 noise and speech (DNS Challenge), with controlled damage density. One axis is the fraction of corrupted duration, from 5% to 100%, using transient noise bursts, clipping, packet dropouts and short bandwidth cuts. A second axis is stationary noise at several SNRs. Standard DNS 2020 synthetic and real sets, VoiceBank-DEMAND, and EARS-WHAM and EARS-Reverb provide comparability with published numbers.

The metrics are as follows:

  • NISQA, PESQ and POLQA for quality.
  • SI-SDR, computed separately on undamaged and damaged frames.
  • Word error rate from a frozen recognizer, together with a hallucination rate counting word substitutions not present in the clean transcript.
  • Speaker-embedding similarity.
  • Fraction of tokens regenerated, and wall-clock time.
  • A small MUSHRA-style listening test on the hardest condition.

Four checks are decisive.

The first is a curve of SI-SDR on undamaged frames and total compute against damage density, for gated versus global regeneration. Gating should dominate at low density and converge to global at full density without loss.

The second compares hallucination rate across critic variants at matched compute. This is where the external critic must beat self-confidence remasking.

The third is detector recall versus end quality, using oracle labels as the ceiling. Missed spans are the known failure of detect-then-edit systems, so we report how far performance falls from the ceiling.

The fourth checks that blending does not explain the fidelity gains. If gating beats global regeneration on SI-SDR but not blending, the localization story is incomplete. If the external critic fails to beat self-assessment, the verification story is wrong. Either result would be published as the finding.