Modern text-to-speech systems often produce an utterance that is almost right: the words are correct and the voice is right, but one word carries the wrong stress or one pause falls in an odd place. Automatic metrics such as word error rate miss these flaws, while listeners notice them at once. Current fixes treat the generator as a black box. They either throw the whole utterance away and resample it, or they hand the finished audio to a second model that edits it. This proposal borrows an idea from fault-tolerant distributed systems: record a lightweight log of the computation while it runs, so that after a failure you replay only the damaged part, starting from the right point in the computation.
The target is diffusion and flow-matching TTS models. These generators build speech in many small refinement steps, starting from noise and ending at a clean spectrogram. We log the model's running estimate of the final spectrogram at a sparse set of those steps. When a perceptual critic flags a local defect, we look back through the log to find the step at which the defect first became fixed in the model's estimate. We then roll back only the affected time span to just before that step and regenerate it, while the rest of the utterance stays clamped to its logged trajectory. The claim is that the depth of the rollback, meaning how many refinement steps we undo, determines whether a repair works. If the rollback is too shallow, the defect survives. If it is too deep, correct prosody and speaker timbre are destroyed. The model's own logged trajectory, read by a perceptual detector rather than a signal-level one, tells us the right depth.
Core Claim
A logged generation trajectory provides two things that no post-hoc repair method has. The first is provenance: the earliest refinement step at which a perceptually detected defect appears in the model's intermediate predictions. The second is the exact internal state at that step for every frame outside the defect. Rolling back to the provenance step and resampling only the defective span should fix the defect more often than resampling from scratch, more cheaply than best-of-N selection, and with less speaker and prosody drift than editing the finished audio. Provenance must be read with a perceptual detector. Signal-level change measures, such as pitch-track or mel distance between successive estimates, should choose the wrong depth, because they diverge from human judgment in the ways documented for generative audio (ViSQOL v3, SongEval).
Relation to Prior Work
Three systems papers supply the mechanism. Swift recovers from crashed workers during distributed training. It logs activations only at group boundaries and recomputes the steps in between, which is 57.3% to 76.3% faster than checkpoint-restart for pipeline-parallel models. Dependency Logging writes the dependency graph between transactions into the database log, so that after a crash only dependent work is replayed serially and independent work runs in parallel.
We reuse two ideas from these papers. From Swift we take boundary logging with recomputation: with a deterministic sampler and a fixed seed, any intermediate step can be recomputed from the nearest logged boundary. From Dependency Logging we take replay restricted to dependent units: only the defective span is regenerated. Both papers define failure as a hardware crash with an exact correct state to restore. Our failure is perceptual, and there is no single correct state to restore. Swift also relies on undoing optimizer steps exactly by inverting them. That option is unavailable here, because stochastic sampling and quantized codecs are not invertible. This constraint is why we log the trajectory rather than try to invert it.
Archer is the closest generative analogue. It caches prompt-side hidden states in diffusion language models so that revising tokens does not force full recomputation. It refreshes the cache when the Hamming distance between the current and cached generations exceeds a fixed radius, and it reports a 2.57x mean speedup. Archer leaves response-side caching open, evaluates only on text and code, and uses a symbolic invalidation rule. We cache and roll back the response itself, in continuous audio. Our rollback decision comes from a perceptual detector, not a symbolic distance.
The perceptual side motivates the problem. LoopTTS uses an AudioLLM judge (Gemini-3-Pro) to diagnose prosodic defects in finished audio, then passes the audio to a separately trained refiner model. Listeners preferred its repairs over raw defective audio 85.7% of the time and over regeneration from scratch 77% of the time. It also raised MOS on flagged neutral utterances from 3.21 to 4.17. LoopTTS shows that local repair beats resampling. However, it keeps no record of how the audio was generated, it requires a second trained model, and its authors note that changing expressiveness tends to lower speaker similarity.
SpeechJudge and MuseCritic show that critics which reason before scoring beat plain scalar regressors. SpeechJudge-GRM reaches 77.2% on SpeechJudge-Eval, against 72.7% for a Bradley-Terry scalar model. These critics also share a stated weakness: they score whole utterances and are slow. This constrains our design. We use the slow critic only to decide whether to repair. We distill it into a fast frame-level detector that decides where the defect is and when in the trajectory it began.
Research Program
We start from an open flow-matching zero-shot TTS model, the kind of system among the six architectures used to build SpeechJudge-Data. We run it with a deterministic ODE sampler and a fixed seed. During normal generation we log the predicted clean mel-spectrogram, together with the solver state, at every k-th step. Following Swift's group-boundary logic, k is chosen so that any unlogged step can be recomputed exactly from its boundary. A 10-second utterance at 32 steps costs a few megabytes, which is negligible next to model memory.
The defect detector is a small temporal network that runs over mel frames and outputs a per-frame defect probability. We train it on two sources of labels. The first is the stress and pause difference labels in REFINER-DB (~42,000 contrastive tuples from LoopTTS), mapped to frames by forced alignment. The second is word-level defect locations elicited from an AudioLLM judge on generated utterances. Because the detector must read intermediate estimates as well as finished audio, we also train it on logged intermediate predictions, labeled with the defects that later appear in the final output.
Repair proceeds in four steps. First, the utterance-level filter and judge decide whether to repair, as in LoopTTS. Second, the detector localizes the defect span on the final output. Third, the detector is run backward over the logged predictions for that span. The provenance step is the earliest step at which the defect probability crosses a calibrated threshold, and we roll back one logged boundary earlier than that. Fourth, the span, plus a short crossfade margin, is resampled from that step with a fresh seed, while all other frames are clamped to their logged trajectories at every step. Optionally, several resamples of the span run in parallel and the detector picks the best one. This is span-local best-of-N, and its cost is a fraction of whole-utterance best-of-N.
Baselines
The naive reference point is to do nothing and keep the defective output. Its partner is regeneration from scratch with a new seed. LoopTTS reports that its refiner is preferred over regeneration in 77% of comparisons, so a method that cannot beat regeneration clearly is not worth pursuing.
The strongest existing repair method is the LoopTTS refiner (LoopTTS). Its reported numbers are 85.7% preference over raw defective audio, 77% over regeneration, and a MOS gain from 3.21 to 4.17 on flagged neutral speech. If its refiner checkpoint is not available, we will reimplement it on REFINER-DB with the same judge. We will report that we did so, and we will not reuse the published figures as if they came from our setting. Beating it means a statistically significant blind A/B preference for our repair over the refiner on the same flagged utterances, with speaker similarity no worse.
The strongest compute-matched competitor is best-of-N reranking with SpeechJudge-GRM as the selector. We give it exactly the same number of model function evaluations (NFE) as our method, including our span resamples, and the same critic calls. The file gives no reported best-of-N quality numbers for TTS (no reported result found on Lacuna for best-of-N speech repair). We therefore run the comparison ourselves at matched NFE values from 1x to 8x.
The most serious baseline is post-hoc partial re-noising. Here the finished audio's defective span is re-noised to some intermediate step and regenerated with the same model, with no log. We run this at every fixed depth in a sweep. We also run it with an oracle depth chosen per utterance using the human-preference judge. This baseline defines the two ablations that isolate our key ingredients. Replacing the logged state with the re-noised final output at the same depth isolates the value of logging. Replacing the provenance-selected depth with a fixed depth isolates the value of provenance. A third ablation replaces the perceptual detector with a signal-level change detector (pitch-track and mel distance between successive estimates) to choose the depth. This tests whether the perceptual-objective gap matters for the rollback decision.
All methods share the same base model, the same flagged utterances, the same judge, and the same tuning budget. Every hyperparameter, including threshold, crossfade length and fixed depth, gets its own small development-set sweep.
Why not just…?
Why not just regenerate from scratch? LoopTTS already shows listeners prefer local repair over regeneration in 77% of comparisons. Regeneration also tends to introduce new defects elsewhere and to drift the speaker identity. We include it mainly as a floor and to measure how often the original defect recurs, which happens when a model systematically glitches on a given phrase.
Why not just take the best of N full samples? Best-of-N costs N full generations. If the defect is systematic, every candidate may carry it, a weakness SpeechJudge itself notes. Our compute-matched comparison settles this directly. If best-of-N at equal NFE matches our repair, the logging machinery is not worth its complexity, and we will say so.
Why not just re-noise the finished audio and inpaint the span, with no log at all? This is the real threat. A well-chosen fixed re-noising depth may capture most of the gain. The oracle-depth and same-depth ablations are designed to measure this. If our method beats fixed-depth re-noising but not oracle-depth re-noising, then the contribution reduces to choosing the depth, and the log is useful mainly as a cheap way to compute provenance. That is still a result, but a narrower one. If logged states beat re-noised states at the same depth, the exact-trajectory claim holds. We expect this mainly for speaker similarity and boundary smoothness, because re-noising the finished audio cannot recover the model's own internal state for the frames around the span.
Why not just train a better refiner, as LoopTTS does? A refiner needs a separate trained model and a contrastive dataset. LoopTTS also reports trade-offs between expressiveness and speaker similarity. Our method needs only the base model and a small detector, so a tie at lower training cost would already be useful. Either result is informative.
Proposed Experiments
We generate 5,000 or more utterances from the base model, using regular and expressive text and reference voices in the style of SpeechJudge-Data. We keep those flagged as defective by the LoopTTS-style filter and judge, and hold out a set for evaluation.
The primary outcomes are blind A/B human preference and MOS on flagged utterances. We split results into neutral and emotional speech, mirroring the categories LoopTTS reports. Secondary outcomes are the following:
- Pairwise automatic preference from SpeechJudge-GRM, whose 77.2% agreement on SpeechJudge-Eval bounds how much it can be trusted.
- Defect survival rate, measured as the judge's re-check of the same span.
- Word error rate.
- Speaker similarity.
- A boundary-discontinuity measure at the edges of the span.
- Cost, measured as NFE and wall-clock time, plus logging memory overhead and its effect on generation latency.
Three checks decide whether the core claim holds. The first is the depth curve. We plot repair success against rollback depth for each defect type. The provenance-selected depth should sit near the per-utterance oracle optimum, and both fixed shallow depths and from-noise inpainting should fall off on either side. If the curve is flat, depth does not matter and the core claim fails. The second check is logged versus re-noised states at equal depth, which shows whether logging adds anything beyond depth selection. The third check is depth chosen by the perceptual detector versus depth chosen by the signal-level detector. We predict the signal-level detector rolls back too deep on harmless spectral changes and too shallow on prosodic errors that produce little spectral change. That would carry the perceptual-objective divergence from evaluation into the internals of generation control.
If all three checks hold for speech, a follow-up applies the same procedure to singing voice synthesis on GTSinger, with code at AaronZ345/GTSinger. There, rhythm and pitch-technique defects let us test whether provenance generalizes beyond stress and pauses.
References
- ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metric
- SongEval: A Benchmark Dataset for Song Aesthetics Evaluation
- Swift: Expedited Failure Recovery for Large-Scale DNN Training
- Scaling distributed transaction processing and recovery based on dependency logging
- Archer: Adaptive Reuse of Cached Hidden States for Efficient Rollback in Diffusion Language Models
- Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correction
- SpeechJudge: Towards Human-Level Judgment for Speech Naturalness
- MuseCritic: Learning Multi-Aspect Song Rewards through Natural-Language Aesthetic Critiques
- GTSinger: A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Tasks