Refine Only Where the Phones Are Uncertain: Sparsity-Gated Speech Editing

Text-based speech editing means changing a few words in a recording by editing its transcript. The best current editors are slow because they regenerate audio by repeated refinement. A diffusion model, which starts from noise and cleans up an output over many passes, runs every pass over every frame in the edited span, even though most of those frames stop changing after the first pass or two. This proposal moves the refinement out of audio space and into a sparse phonetic posteriorgram. A phonetic posteriorgram (PPG) is a per-frame probability distribution over phonemes. A sparse PPG keeps only the few most likely phonemes in each frame and sets the rest to zero. In that space, a frame inside a steady vowel is almost one-hot and needs no further work. The frames that stay ambiguous sit at phone boundaries, at the seams of the edit, and in regions of coarticulation (where neighboring sounds blend into each other). The number of nonzero entries in each frame tells the system where to keep spending compute.

The core proposal is a coarse-to-fine editor that works in three stages. It first settles word durations, then phone durations, then frame-level sparse posteriors. At each level it freezes the units whose sparse distribution has collapsed to a single confident class. Later passes run only on the frames that remain uncertain. A single-pass vocoder (a network that turns acoustic features into a waveform), conditioned on the sparse PPG plus pitch and loudness, then produces the audio. If the idea holds, refinement cost grows with how much of the edit is phonetically uncertain, rather than with utterance length times step count. The sparse bottleneck should also make the editor less sensitive to the recording conditions of the training data.

Core Claim

Two existing systems motivate the design. Fine-Grained and Interpretable Neural Speech Editing sparsifies each frame's PPG by keeping only its top-percentile entries (percentile-k sparsification). It pairs this with pitch, periodicity and loudness to drive a single-pass vocoder, and reports that sparsity removes speaker and channel leakage. It has no generative refinement and no structure above the frame level, and its own known weakness is hard cutoffs at phone boundaries. FluentEditor2 shows that frame-, phoneme- and word-level smoothness losses matter for fluent edits. However, it diffuses dense Mel-spectrograms, which are slow to sample, and it degrades on unseen domains such as LJSpeech.

The claim is that these two weaknesses fit together. In sparse-PPG space, the frames that need iterative repair are the same boundary frames where hard sparsification produces artifacts. The width of each frame's sparse support is a cheap stopping signal that a refiner can use to decide which frames to keep working on.

Concretely, let $A_t$ be the set of uncommitted frames at refinement step $t$, and let $N$ be the number of frames in the edit region. A uniform refiner with $T$ steps costs $T \cdot N$ frame updates. The gated refiner costs $\sum_t |A_t|$. A frame leaves $A_t$ once its sparse support is a single class above a confidence threshold and that class did not change on the previous step. The empirical question is how fast $|A_t|$ shrinks on real edits, and whether frozen frames ever need to be reopened.

Research Program

The editor has three parts.

The first part is a phonetic front end. It is a phoneme recognizer trained on aligned English speech, whose posteriors are sparsified as in the Interspeech 2024 editing paper. Together with pitch and loudness extracted from the unedited audio, these posteriors form the representation of the original recording.

The second part is a non-causal Transformer refiner, which is a model that sees the whole sequence at once. Its inputs are the target phone sequence from the edited transcript, the frozen context on both sides of the edit, and a current draft. It outputs updated sparse posteriors, pitch and loudness for the edit region. The refiner works at three levels. At the word level it predicts durations for the edited words and freezes any word whose duration estimate is stable. At the phone level it refines phone durations only inside unfrozen words plus a small window into their neighbors. At the frame level it refines sparse posteriors and prosody only on frames in $A_t$. Committed frames keep their cached attention states, so each pass processes only the active frames.

The design borrows from Align-Refine, which iteratively edits CTC alignments (CTC is a training method that aligns each audio frame to an output symbol or a blank). Align-Refine refines every frame at every step and works on hard symbols. This system instead refines soft sparse distributions and stops each frame individually.

Errors at a higher level can feed into the levels below. To limit this, a frame-level failure can reopen its parent phone for one additional pass. A failure here means a frame whose support grows instead of shrinking across two steps. The proposal will count how often reopening fires.

The third part is a single-pass vocoder conditioned on the sparse PPG, pitch and loudness. Its latency is counted in every timing number. VocalNet-MDM identifies vocoder latency as the remaining bottleneck after its own decoding speedup.

Training uses masked-span reconstruction on VCTK and LibriTTS. A model trained on GigaSpeech is added for the in-the-wild benchmark.

The analysis is as much a deliverable as the system. It covers three things. The first is the distribution of commit steps per frame, broken down by frame type: phone interior, phone boundary, edit seam, and silence. The second is whether frames that commit early ever turn out wrong. The third is how the sparsity level $k$ trades boundary smoothness against leakage and against the size of the active set.

Baselines

The naive reference point is cut-and-splice. It synthesizes the new words with a text-to-speech model conditioned on the speaker, then concatenates them into the recording. This shows how much any learned editor buys. The second reference point copies the single-pass sparse-PPG approach of Fine-Grained and Interpretable Neural Speech Editing. The edit region's sparse PPG is predicted in one feedforward pass from the target phones and then vocoded. This is the "zero refinement" point on our curve.

The strongest dense competitor on VCTK and LibriTTS is FluentEditor2. It reports Mel-Cepstral Distortion (MCD) of 5.836 and fluency opinion scores (FMOS) of 4.314 for insertion and 4.286 for replacement on VCTK, and MCD of 4.537 on LibriTTS (FluentEditor2). We will retrain it on our exact splits with its published step schedule. We will also run it at reduced step counts so that it can be compared at matched wall-clock time.

The strongest competitor on realistic edits is HiCoDD from Multi Codec Discrete Diffusion Model for Text Guided Speech Inpainting and Editing. It is a hierarchical discrete diffusion model over the codebooks of a neural audio codec. It reports word error rate (WER) of 0.121 and speaker similarity of 0.98 on RealEdit using 512 denoising steps. No reported real-time factor for it was found on Lacuna, so we will measure it ourselves on the same GPU. VoiceCraft, an autoregressive codec language model, is the autoregressive reference. It reports that its RealEdit edits were preferred over the original recording 48% of the time.

The ablation that isolates the key ingredient is our own refiner with gating turned off. It refines all frames for $T$ steps and is compared with the gated refiner at equal total frame updates. A second ablation removes the hierarchy and gates only at the frame level.

Fairness rules are the same for all systems: identical training data per benchmark, an equal hyperparameter search budget, the same hardware, end-to-end timing including the vocoder, and the same ASR model scoring WER. Beating the dense baseline means matching FluentEditor2's MCD and FMOS within statistical noise at a lower real-time factor. Beating the strongest baseline means matching HiCoDD's RealEdit WER and speaker similarity at an order of magnitude fewer network evaluations.

Why not just…?

Why not just distill a dense or codec diffusion editor to one to four steps? This is the real threat. VocalNet-MDM compresses masked diffusion to 1–4 steps through self-distillation, reaching a real-time factor of 0.0116. However, it does so for streaming dialogue tokens, not for editing, and its authors note that the trade-offs become severe at very low step counts. Distillation and gating are orthogonal, so we will apply the same self-distillation recipe to both the dense refiner and the gated sparse refiner. If the distilled dense editor matches our quality at equal latency, the latency argument fails. The project would then reduce to a claim about robustness from the sparse bottleneck, and we would report it that way.

Why not just use self-supervised or codec tokens instead of sparse PPGs? These tokens do not separate content from timbre and recording channel. Treating codec levels as one flat sequence raised RealEdit WER from 0.139 to 0.186 in the HiCoDD ablation. VoiceCraft reports silences and scratching artifacts that required repeated resampling. The sparse PPG has a second advantage: its support width is an interpretable per-frame stopping signal, which dense tokens lack. We will test this directly by applying the same gating rule using the entropy of a codec-token refiner's predictions.

Why not just fewer uniform steps, or a one-shot model? Truncating steps uniformly cost WaveGrad 2 quality, dropping from 4.43 to 4.32 MOS when moving from 1,000 to 50 steps (WaveGrad 2). One-shot non-autoregressive models suffer from the inconsistency between independently predicted frames documented by Align-Refine. Both cases appear in our compute-versus-quality curve as the ungated ablation and the zero-step baseline. Gating has to sit strictly above them at every compute budget to justify itself.

Proposed Experiments

The first decisive check is a matched-compute curve. On VCTK and LibriTTS insertion and replacement edits, we plot MCD, FMOS, and ASR WER against total frame updates and against end-to-end real-time factor. The curves compare gated, ungated, and single-pass variants. The hypothesis fails if gating does not dominate at equal compute.

The second check is error localization. If frames committed in the first step account for a meaningful share of perceptual errors, the stopping signal is wrong, and the threshold or the reopening rule must change.

The third check is cross-domain transfer. FluentEditor2 reports a drop on LJSpeech, but no reported number for that drop was found on Lacuna. We therefore train all systems on VCTK and LibriTTS and evaluate on LJSpeech and DAPS. The test is whether the sparse bottleneck shrinks the gap between in-domain and out-of-domain scores relative to the retrained FluentEditor2.

The fourth check uses RealEdit with a GigaSpeech-trained model. We compare against HiCoDD and VoiceCraft on WER, speaker similarity, and a human preference test, and report measured latency for every system.

Throughout, we also sweep the sparsity level $k$. Too small a $k$ should reintroduce boundary artifacts. Too large a $k$ should inflate the active set and reintroduce leakage. Locating that trade-off is a result in its own right.