Automatic speech-quality scores are computed for a whole clip. A synthesized sentence with one misplaced stress or one unnatural pause can score almost as well as a flawless one, even though a listener hears the flaw at once. This proposal tests a simple claim. A learned quality predictor trained only on whole-clip ratings may still respond sharply to the flawed word, even though the flaw barely moves its averaged score. If so, we should read out that response instead of the score. Concretely, we take the gradient of the predictor's output with respect to its internal frame features, summed per word, and ask whether it points at the defective word. If it does, we get a cheap, local, label-free defect detector. It would replace the slow commercial audio-language-model judge that current correction systems use to find which word to fix.
The opportunity matters because the closest working system already shows that fixing localized defects pays off. Diagnose, Then Refine (LoopTTS) asks Gemini 3 Pro to name defective words. It then resynthesizes with a loss weighted toward those words, and listeners preferred the result over the defective audio 85.7% of the time. Its weak point is the step that finds the defect. That step is a black-box prompt to a large external model, it adds substantial latency, and it emits only coarse instructions. Our question is narrow and testable: can a single backward pass through an open quality predictor do that localization step about as well?
Core Claim
Gradient attribution measures how much a model's output changes when one part of its input is nudged. Applied to an utterance-level quality predictor such as UTMOS (a mean-opinion-score predictor built on a self-supervised speech encoder), the claim is that it localizes perceptual defects at the word level well enough to drive targeted resynthesis. It should also outperform cheaper readouts of the same predictor, namely its pre-pooling frame scores and word occlusion.
The key ingredient is the reference point for the attribution. Path-based attribution such as Integrated Gradients accumulates gradients along a straight line from a "null" reference input to the real input. The file notes that a poorly chosen reference injects out-of-distribution artifacts. We replace the usual zero reference with a content-matched one: a neutral rendering of the same text, time-warped word by word onto the input's word boundaries using forced alignment. The interpolation then happens in the predictor's encoder-feature space.
With this reference, attribution answers "which words make this rendering sound worse than a plain rendering of the same sentence." That is the question a correction system needs answered. A zero reference instead answers "which frames carry any signal at all," which is mostly loudness and speech presence. Formally, with $h$ the input's word-aligned encoder features, $h'$ the warped neutral reference, $F$ the predictor's scalar score, and $W_w$ the set of frames inside word $w$, the word score is
$$A(w) = \sum_{t \in W_w} (h_t - h'_t) \cdot \int_0^1 \nabla_{h_t} F\big(h' + \alpha (h - h')\big)\, d\alpha .$$
The integral is approximated with roughly 16 to 32 steps. The predicted defect location is the word with the most negative $A(w)$.
We also test a second target, borrowed from Localize and Neutralize. That paper showed something useful about vision-language attacks: when the output looks benign, gradients of the output are blind to the attack, but the gradient of an early hidden-state norm still finds the offending tokens. A high-scoring clip with one bad word is the speech analogue of "benign output." We therefore attribute both the pooled score and the norm of the predictor's first post-encoder hidden state.
Relation to Prior Work
Three papers supply the method. Localize and Neutralize localizes sparse harmful tokens in vision-language models with one forward-backward pass, adding about 0.22 seconds. Detecting, Explaining, and Mitigating Memorization in Diffusion Models differentiates an internal magnitude to find trigger tokens in Stable Diffusion. Similarizing the Influence of Words uses gradient attributions to split text into influential and uninfluential words. All three work on discrete text or image tokens and target security or memorization. None of them touches continuous audio or perceptual quality. Our contribution is to move this one-pass localization into speech evaluators and to add the content-matched reference, which has no counterpart in those settings.
Two papers motivate the gap. LoopTTS documents that word error rate and mean UTMOS miss sporadic prosodic defects. SongEval shows that a UTMOS-style model retrained on aesthetic ratings reaches Pearson 0.912 with human song ratings, against 0.61 to 0.66 for Audiobox-Aesthetic. Learned predictors of this family therefore carry real perceptual signal; the question is whether that signal is spatially readable.
One paper constrains what we may claim. Do Music Generation Models Encode Music Theory? shows that probes can find information in hidden states, but that turning such information into intervention remains unsolved. We therefore validate localization only by downstream correction and listener judgments, never by attribution maps alone.
A further caution comes from Improving Complex Reasoning with Dynamic Prompt Corruption, which reports that attention weights misdiagnose distracting tokens. We include attention as a baseline rather than assume it fails.
Research Program
We first build a test set with known defect locations. A reliable ground truth is needed because REFINER-DB's word labels, about 42,000 contrastive tuples, were produced by Gemini 3 Pro. Scoring Gemini against them would be circular.
The test set has three layers. The first is synthetic injection into clean LibriTTS and ESD utterances: one word per clip receives a misplaced-stress edit (energy and pitch reshaping), a flattened pitch contour, a spurious pause, or a duration distortion, and the edited word is recorded. The second is a human-verified subset of about 1,000 REFINER-DB tuples, on which two annotators mark the defective word independently of Gemini's label. The third is a transfer layer built from GTSinger singing (dataset), which provides phoneme-level alignments. We inject the same edit types there, plus pitch slips against the MusicXML score. Here no speech-trained supervised localizer has seen the domain.
We then compute word scores from three predictors: off-the-shelf UTMOS, UTMOS fine-tuned on mean-opinion ratings, and a SongEval-trained predictor for the singing layer. Each predictor is scored under both targets (pooled score and hidden-state norm) and both references (content-matched and zero). Finally, we feed the top-ranked word or words into a LoopTTS-style refiner as its position weights, replacing Gemini's localization. We reuse the LoopTTS refiner if it is released and otherwise reimplement its position-weighted loss. A listening test compares the outputs.
Baselines
The naive reference points are a random word chosen as the defect location, and, for clip-level ranking, the pooled UTMOS score itself. The pooled score sets the "do nothing" level for whether a localized readout recovers preferences that the clip score misses. Every attribution variant must clearly exceed random top-1 word accuracy on the synthetic layer.
The strongest cheap competitor uses the same predictor. UTMOS-family models output per-frame scores and then average them, so the most obvious localization is to take the minimum of those frame scores, aggregated per word. It costs nothing extra and needs no gradients. It is run on identical clips and alignments. Beating it means higher top-1 and top-3 word accuracy on the synthetic and human-verified layers, by a margin outside the bootstrap confidence intervals.
The second competitor is word occlusion. Each word's frames are replaced, one word at a time, with the corresponding frames of the content-matched neutral reference, and the predictor rescores the clip; the word whose replacement most raises the score is the prediction. This is a perturbation method that asks the same counterfactual question as our attribution, at a cost of one forward pass per word. Because it shares our reference, it isolates whether gradients add anything beyond the reference. We match its accuracy and report wall-clock time. Beating it means comparable or better accuracy at lower latency, or better accuracy on relational defects such as stress, where the right answer depends on context.
The third competitor is attention rollout over the predictor's transformer layers, run on the same clips.
The strongest existing system is the LoopTTS Gemini 3 Pro judge. The reported numbers for its full pipeline are 85.7% human preference over defective audio, 77% preference over regenerating from scratch, and mean opinion score improvements from 3.21 to 4.17 on neutral speech and from 3.01 to 4.01 on emotional speech (LoopTTS). No reported result was found on Lacuna for Gemini's word-level localization accuracy in isolation or for its per-clip latency. We therefore measure both on the human-verified layer under the same prompt LoopTTS describes. Beating it means non-inferiority in the end-to-end listening test (preference within five points of Gemini-localized refinement on matched sentences) together with at least an order-of-magnitude lower localization latency on one GPU.
A supervised word-level defect classifier trained on REFINER-DB's labels serves as an in-domain ceiling, not as a competitor we expect to beat.
Two ablations isolate the key ingredient. The first swaps the content-matched reference for a zero reference. The second swaps the hidden-state-norm target for the pooled score, which tests whether the "benign output" blindness reported for vision-language models appears in speech evaluators.
Why not just…?
Why not just read the predictor's frame scores before pooling? This is the most serious threat, and the experiment is built to settle it. Frame scores reflect what each frame looks like within the encoder's context. A misplaced stress may be locally well-formed, so that its frame score looks fine, and become wrong only relative to the rest of the sentence. The gradient captures how that frame changes the whole-clip judgment. If frame-minimum localization matches attribution on stress and pause defects, then the gradient machinery is unnecessary and we will report that result.
Why not just occlude words and rescore? For a twenty-word sentence, occlusion costs twenty forward passes, which is cheap, so occlusion could plausibly match us in accuracy. We share its content-matched reference precisely so that the comparison isolates the gradient. If occlusion ties, the contribution shrinks to the reference design plus a latency advantage that matters only for long-form audio such as full songs. We will state this plainly.
Why not just train a defect localizer on REFINER-DB? In domain, it will probably win, because it is trained on the task. Its labels inherit Gemini's judgments, however, and it has no path to domains without word-level labels. The GTSinger transfer layer is where this comparison is decided. If the speech-trained localizer transfers to singing as well as attribution from a song-quality predictor does, then the label-free argument fails.
Why not just keep the Gemini judge? CMI-RewardBench reports that Gemini 3 Pro reaches only 65.80% agreement with experts on music musicality pairs, and LoopTTS acknowledges that its latency is substantial. Gemini's speech localization may nonetheless be strong. We therefore compare against it on independently verified labels and in the listening test, rather than dismissing it.
Decisive Checks
The first check is word-level localization accuracy on the synthetic layer, broken down by defect type, for content-matched Integrated Gradients against frame-minimum scores and occlusion. If attribution fails to beat frame scores on stress and pause defects, the core claim is false.
The second check covers clip pairs that differ only by one injected defect. Here we measure how often the worst-word attribution score ranks the clean clip above the defective one, compared with the pooled score. This directly measures how much of the gap between metric and perception a localized readout closes.
The third check is the end-to-end listening test, comparing attribution-localized refinement with Gemini-localized refinement on 300 held-out sentences, with latency recorded.
The fourth check repeats the first on GTSinger singing, using the SongEval-trained predictor. It determines whether the method reaches beyond speech, which is the only place where it could do something supervised localizers cannot.
References
- Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correction
- Localize and Neutralize: Gradient-Guided Token Suppression Against Visual Prompt Injection Attack
- Detecting, Explaining, and Mitigating Memorization in Diffusion Models
- Similarizing the Influence of Words with Contrastive Learning to Defend Word-level Adversarial Text Attack
- SongEval: A Benchmark Dataset for Song Aesthetics Evaluation
- Do Music Generation Models Encode Music Theory?
- Improving Complex Reasoning with Dynamic Prompt Corruption: A Soft Prompt Optimization Approach
- GTSinger: A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Tasks
- CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction