Where to Copy and Where to Generate in Speech Restoration

Generative speech enhancers no longer filter noise out of a recording. They treat the noisy recording as a hint and re-synthesize clean speech from a learned model of what speech sounds like. Listeners prefer the results, but these systems sometimes produce fluent, natural-sounding words that the speaker never said. This failure is called phonetic hallucination. Novel view synthesis in computer vision had a structurally similar problem and recently found a clean fix. Some output pixels have a ground truth visible in the source image, so they should be copied. Other pixels were hidden in the source and only become visible from the new viewpoint (they are "disoccluded"), so they must be generated. Models that apply one loss and one generator to both kinds of pixel either blur the hidden regions or alter the visible ones.

This note proposes treating speech restoration as the same split. Time-frequency regions where speech clearly dominates the noise are "observed" and should be copied from the input along a path that never passes through the compressed token space of a neural codec. Regions buried under noise are "disoccluded" and are the only places where the generative prior may act. The proposal's core empirical bet is specific: a meaningful share of hallucinated words in today's generative enhancers occurs where the evidence was actually present. The prior overwrites the evidence because everything is routed through a lossy codec bottleneck. If that is true, much of this hallucination can be removed without giving up the naturalness that makes generative restoration attractive.

Core Claim

The closest existing work is GenRec. It uses projected monocular depth to build an "observation mask" for sparse-view 3D synthesis, and it pairs a latent flow-matching generator with a pixel-space branch. That branch pulls high-frequency detail from source images and only touches observed pixels. Its gradients are isolated so that regression does not dull the generator. GenRec works only on static images.

We carry its three ingredients into generative speech restoration, where they have not been applied:

  • a mask separating observed from unobserved content,
  • a copy path that bypasses the latent bottleneck,
  • gated losses with isolated gradients.

The new claim is diagnostic as well as architectural. Phonetic hallucination has two distinct sources, and they need different remedies.

  • Overwriting: observed phonetic evidence is rewritten after passing through a lossy codec such as the Descript Audio Codec (DAC) tokens in MaskSR2 or the 2 kbps codebooks in Audio-Visual Speech Codecs. Copying fixes this.
  • Gap-filling: the prior invents content in spans where evidence is genuinely absent. Copying cannot fix this. Only stronger side evidence can, such as lip motion, or an honest uncertainty signal.

Current systems treat both as one problem. Examples are semantic distillation in MaskSR2 and visual conditioning in AV-UDiffSE+. A third source, borrowed from GenRec's depth-error failure, is mask error. Noise misclassified as observed speech would be faithfully copied. This is a plausible route to the "babbling" failure documented by Serrà et al..

Research Program

Step 1: attribution study. The first piece of work is measurement, and it is useful whether or not the method succeeds. We take existing generative restorers and align each word or character error to its time span:

  • a masked-token restorer in the MaskSR2 style;
  • the flow-matching audio-visual model FlowAVSE;
  • a score-based diffusion enhancer in the line of Richter et al..

We then label each error span by the fraction of its time-frequency energy that an oracle mask, computed from the clean and noise stems, marks as speech-dominated. If many errors fall in mostly-observed spans, overwriting is real and the copy path is justified. If almost all errors fall in noise-dominated spans, the copy path will help naturalness and fidelity but not hallucination. In that case the program pivots toward the disoccluded regime, where lip evidence is the only remedy.

Step 2: architecture. Let $Y$ be the noisy complex spectrogram, $M \in [0,1]$ an estimated soft observation mask, $\Delta$ a small residual correction predicted by a lightweight spectrogram-domain network, and $G$ the decoded output of the generative model. The restored spectrogram is

$$\hat{S} = M \odot (Y + \Delta) + (1 - M) \odot G,$$

with a short learned smoothing at mask boundaries. The copy term is the audio counterpart of pixel warping. It cannot invent content, only correct the gain and phase of what is already there. This is the same reason Coordinate-Based Texture Inpainting predicts source coordinates rather than colors.

Step 3: training. Training separates the two objectives. A regression loss applies only to observed bins and updates only the copy network. The generator's own objective (token cross-entropy or flow matching) is weighted by $1 - M$. This avoids the "uniform loss" conflict GenRec identifies: regression pulls the prior toward averages, and generation perturbs known content. The generator still conditions on the full input, and on lip features where video exists, so its output in gaps stays coherent with what it can see.

Step 4: the mask itself. We estimate $M$ with a discriminative time-frequency network and calibrate it against oracle masks. In the audio-visual setting, we lower $M$ wherever lip-motion features from AV-HuBERT disagree with the acoustic evidence. We study sensitivity by injecting controlled mask errors in both directions, mirroring GenRec's depth-failure analysis.

Baselines

Naive reference. The naive reference is the unprocessed noisy input. It is joined by the strongest discriminative system described in the Interspeech 2025 URGENT Challenge, a large sub-band recurrent network (team T1). It won on intrusive scores and character accuracy and did not hallucinate on unseen Japanese. It defines the fidelity ceiling we must approach while keeping generative naturalness. No reported numeric CAcc for T1 was found on Lacuna, so we will retrain a comparable sub-band model on the same data mixture and measure it ourselves.

Strongest generative baseline (audio-only). This is MaskSR2, which distills HuBERT semantics into a DAC-token predictor. It reports 3.18% word error rate (WER) on full-band VCTK for its large variant, versus 4.74% for MaskSR-L, with a subjective MOS of 4.72 against 4.76 for clean speech. Beating it means lowering WER below 3.18% on that test without a MOS drop beyond the confidence interval.

Strongest audio-visual baseline. This is AV-UDiffSE+. On LRS3 with mismatched noise it reports PESQ 2.42 and SI-SDR 3.67 dB, against PESQ 1.49 and 3.12 dB for FlowAVSE. On VoxCeleb2, FlowAVSE reports 13.37 dB SI-SDR at a real-time factor of 0.070. We add our gating to a FlowAVSE-sized backbone. The win condition is AV-UDiffSE+-level PESQ on mismatched LRS3 at FlowAVSE-like speed, with lower character error. No reported WER for either system was found on Lacuna, so we measure it.

Fairness and ablations. Every system is retrained or fine-tuned on the identical data mixture with matched parameter count and the same tuning budget, measured in GPU-hours for hyperparameter search. The key ablations isolate the proposal's ingredients:

  • the same backbone with the copy path but uniform losses and no gradient isolation;
  • the same backbone with gated losses but no copy path;
  • the full model with oracle versus estimated masks.

Why not just…?

Why not just use the discriminative T1-style model? It is the real fidelity reference and may stay ahead on character accuracy. But listeners consistently preferred generative systems in URGENT 2025. Listeners also preferred generative re-synthesis over Demucs 73% of the time in Audio-Visual Speech Codecs. Our claim is a better point on the curve, not a dominance. If the gated model cannot match T1's character accuracy within a small margin while keeping a clear MOS lead, the proposal fails, and we will say so.

Why not just blend a discriminative output and a generative output with a mask after the fact? This is the most serious threat, because it is our composition equation without the joint training. We run it directly, using the same mask, the same T1-style model as the copy source, and MaskSR2 as the generator. Our argument against it is that an independently trained generator does not know which regions will be kept. It therefore fills gaps inconsistently with the copied content, producing audible seams and phoneme splits at boundaries. If post-hoc blending matches the full model on WER and MOS, the architectural contribution collapses to this simple recipe. We would report that, and the attribution study would remain the paper.

Why not just add semantic distillation or text and ASR guidance? MaskSR2 already distills HuBERT semantics and still uses a codec that rewrites every frame. Transcripts are unavailable in deployment, and inference-time ASR re-ranking adds latency and inherits ASR's own hallucinations, as with Whisper's phantom words reported by Radford et al.. Semantic distillation is complementary rather than competing. We include a MaskSR2-backbone variant with our gating to test whether the gains add up.

Proposed Experiments

Attribution study. This runs first, on three test sets: full-band VCTK, URGENT 2025 including the unseen Japanese set, and LRS3 under mismatched noise. Its decisive output is the share of character errors falling in spans that are at least 70% observed under the oracle mask. Error transcription uses two independent ASR systems, and only errors both agree on are counted. This is a guard against measuring ASR hallucination instead of enhancer hallucination.

Main comparison. We report:

  • WER or character accuracy, split by span type (observed versus disoccluded);
  • a hallucination rate, defined as inserted or substituted words in spans where the reference has speech that both ASR systems decode correctly from clean audio;
  • intrusive metrics: PESQ and SI-SDR;
  • naturalness: listening-test MOS;
  • real-time factor.

Predicted signature. The method should cut errors in observed spans substantially while leaving disoccluded-span errors roughly unchanged. The URGENT Japanese set is the sharpest test. A prior trained mostly on other languages overwrites unfamiliar phonemes even when they are audible. The copy path should therefore recover much of the CAcc that the generative T13 system lost there.

Mask-error sweep. We bias the estimated mask toward "observed" and toward "unobserved." This should reproduce babbling in the first case and ordinary generative drift in the second, and it tells us which direction of mask error is more costly.

Visual gap test. Within disoccluded spans only, we compare audio-only gating against gating plus lip conditioning. This includes yaw-rotated faces generated with the 3D face-model augmentation of Towards Pose-invariant Lip-Reading. It measures how much of the remaining gap-filling hallucination visual evidence can remove as the mouth turns out of view.