A neural vocoder is the last stage of a speech synthesizer. It turns a compact acoustic description, usually a mel-spectrogram, into the actual waveform. On speech that resembles their training data these models are nearly transparent. On singing, shouting, whispering, accented speech or music they break, and they break in specific ways. Some frames lose track of pitch and sound buzzy. Some produce mirror-image energy above the true harmonics, called imaging or aliasing. Some smear the harmonics together. The field's response has been to redesign the network so a single model makes fewer of these errors everywhere. Once deployed, that model still runs open-loop: nothing checks the output or fixes the few frames that went wrong.
This note proposes to borrow the central lesson of multi-agent task decomposition, which is to separate the worker from independent, specialized checkers. We apply it at inference time to vocoder output. Several small "auditors" each look for one physically defined artifact type in each short stretch of audio, and each is paired with a repair operator suited to that artifact. Every proposed patch then goes to an independent verifier that rejects repairs which create new defects. The claim is concrete and falsifiable. On out-of-distribution audio, vocoder failures are concentrated in a minority of frames and differ in kind. So typed, localized diagnosis and repair should beat both a stronger single vocoder and the obvious shortcut of choosing the best whole-utterance output by a quality score.
Core Claim and What Is New
Two lines of prior work motivate the proposal. Architectural fixes each target one artifact family inside the generator or its training critics. HiFi-GAN adds discriminators that view the waveform at several pitch periods. Avocodo replaces aliasing-prone downsampling with filter banks and adds sub-band discriminators. BigVGAN uses periodic Snake activations with low-pass filtering to keep nonlinearities from creating frequencies above the Nyquist limit. AF-Vocoder moves the anti-artifact work into one learnable frequency-domain filter at the final layer. All four are static. BigVGAN's own music results and VocBench both document the remaining out-of-distribution degradation, and AF-Vocoder reports that some adversarial training setups still leave audible artifacts. The one existing closed-loop audio system, AgentSteerTTS, shows that a listen-critique-recalibrate loop helps speech synthesis. Its critic is a multimodal language model judging whether emotion matches an instruction, and it treats the vocoder as a fixed black box. LVAS-Agent decomposes long-video sound design into planner roles and likewise never inspects the waveform generator.
Our proposal differs from AgentSteerTTS in what the critics look at, how they are built, and what they can change. Its critique is semantic and applies to the whole utterance. Ours is physical and applies to individual frames. It uses one general critic, while we use one narrow auditor per artifact type, each trained from measurable labels. It corrects upstream latents, while we repair the waveform locally with operators matched to the diagnosed failure. We also do not assume the auditors should be language models. The multi-agent literature supplies the structure: role separation, capability firewalls, and an independent tester. It does not require every role to be a language model, and we test whether one helps.
Research Program
We build three pieces.
The first is a set of reference-free artifact auditors. In copy-synthesis, where a real recording is converted to a mel-spectrogram and re-vocoded, the ground-truth waveform exists. That lets us compute frame-level labels for each artifact type directly. Periodicity and voicing errors come from comparing pitch tracks and voiced/unvoiced decisions against the reference. Imaging and aliasing show up as excess energy in bands above the reference harmonics or mirrored around upsampling boundaries. Harmonic blurring appears as reduced harmonic-to-noise contrast relative to the reference. Phase damage is measured by group-delay deviation. We generate such labels at scale by running several vocoders (BigVGAN-v2 from the released codebase, HiFi-GAN, AF-Vocoder if available, and the Neural Homomorphic Vocoder) over LibriTTS plus deliberately stressful audio: singing from GTSinger, vocal stems from MUSDB18-HQ, and emotional speech from ESD. We then train one small detector per artifact type that sees only the synthesized audio. These detectors are the auditors. Each one reports where its artifact occurs and how severe it is. Keeping them separate is the firewall. The pitch auditor is not asked to reason about imaging, so a pitch mistake cannot silently distort the imaging judgment.
The second piece is a set of repair operators, one per artifact type, each using a mechanism from the vocoder literature. Periodicity and voicing failures are repaired by re-synthesizing the flagged segment with the homomorphic vocoder. Its explicit pitch-driven source-filter design makes pitch errors structurally unlikely, and it costs about 15,000 operations per sample, so re-running short segments is cheap. Imaging and aliasing are repaired with a targeted frequency-domain filter applied only to the flagged band and frames, in the spirit of AF-Vocoder's final-stage filter but applied locally. Blurring and generic failures are repaired by re-vocoding the segment with a different network and cross-fading it in. Because of this pairing, the auditors' types matter: a detected artifact determines which fix is attempted.
The third piece is an independent verifier, which plays the "tester" role from Dong et al. and ReChisel. It re-runs every auditor on the patched segment and its boundaries. A patch is accepted only if it reduces the targeted artifact and leaves every other auditor's score no worse, which rules out clicks at splice points or new pitch errors introduced by the replacement vocoder. This addresses the cascading-failure problem that the multi-agent survey by Li et al. identifies. A repair agent that trusts a wrong diagnosis would otherwise make things worse, and the verifier is how we detect that. A scheduler orders repairs when artifacts overlap. We implement it two ways: as fixed rules, and as a language-model coordinator that reads the auditors' reports.
Baselines
The naive reference point is BigVGAN-v2 run once with no post-processing. It is the most widely used universal vocoder, and its released checkpoint was trained for 5M steps on LibriTTS. The strongest reported out-of-distribution figure in our material is its average SMOS of 4.26 on MUSDB18-HQ, against 4.08 for HiFi-GAN (BigVGAN). Beating it means a statistically significant listening-test gain on the same clips and a reduction in periodicity error and voicing errors on the out-of-distribution sets, without loss on LibriTTS test-clean.
The strongest existing architectural method is AF-Vocoder, which reports beating BigVGAN, BigVSAN, Vocos and FreeV on multi-resolution spectral error, PESQ, periodicity error, voiced/unvoiced F1 and UTMOS. The file contains no numbers for these claims, so there is no reported result found on Lacuna for AF-Vocoder's metric values. We will retrain or obtain it on the full LibriTTS training set and evaluate it under our protocol. The comparison is also a composition test: our repair loop should improve AF-Vocoder's output too, and if it does not, the architectural fix already covers what we target.
The most dangerous baseline is utterance-level reranking. We run every vocoder in our portfolio on each utterance and keep the output with the highest UTMOS score. UTMOS is the top-ranked predictor on VoiceMOS 2022, with utterance-level MSE 0.165 and Spearman correlation 0.897 on the main track. This baseline gets exactly the same vocoders, the same compute budget measured in total vocoder calls, and the same held-out data as our system.
To isolate the value of typing, we add a frame-level variant that splices segments from whichever vocoder a single untyped per-frame quality score prefers. Its localization matches our system, but it has no artifact types.
The remaining baselines are ablations. We remove the verifier to measure how often unverified repairs cause regressions. We collapse the auditors into one multi-output detector to test whether separation matters. We replace the rule scheduler with a language-model coordinator. Finally, we give a single multimodal language model the audio and the same repair tools, prompted to diagnose and fix, as the monolithic-critic alternative.
Why not just…?
Why not just use a better vocoder architecture? Architecture is complementary rather than a competitor, and we test that directly by running the loop on top of BigVGAN and AF-Vocoder. The case for inference-time repair rests on the documented residual degradation on music and singing, and on AF-Vocoder's report of lingering artifacts under some training regimes. If our loop yields no gain on AF-Vocoder's out-of-distribution outputs, we will conclude that the architecture has closed the gap.
Why not just add more discriminators? Multi-period and multi-band discriminators shape the generator during training and are then discarded. Avocodo's discriminators raised LJSpeech MOS from 4.150 to 4.258 and reached 4.080 on unseen speakers, but they cannot act on a specific failing frame at deployment. Our auditors are in fact close cousins of discriminators, specialized and kept at inference time. The experiment that separates the two ideas is whether auditor-guided repair improves outputs of vocoders that were already trained with those discriminators.
Why not just pick the best whole output by UTMOS? This is a real threat. If artifacts in a failing utterance are spread across most frames, or if one portfolio vocoder dominates, whole-utterance selection captures almost all the benefit at lower complexity. We therefore measure first how localized the artifacts are, as the fraction of frames flagged per utterance by copy-synthesis labels. We then compare our system to both the utterance-level and frame-level untyped selection baselines at matched compute. If typed repair does not beat frame-level untyped splicing, the typing idea fails. What remains would be a simpler splicing method, and we would report that.
Why not just fine-tune the vocoder on out-of-distribution data? When matched data exists, it is the right fix, and we include BigVGAN fine-tuned on GTSinger and MUSDB18-HQ training splits as an upper reference. Our target is the setting where that data does not exist at deployment. We therefore evaluate on held-out domains excluded from both fine-tuning and auditor training, for example training auditors without singing and testing on singing.
Why not just ask one multimodal language model to find and fix artifacts? The multi-agent literature (MACM, Mouselinos et al.) finds that single models which plan, act and verify at once take shortcuts and fail to self-correct. There is also no evidence in our material that such models can localize a 10-millisecond imaging artifact. We run this baseline with the same tools and score its frame-level diagnostic accuracy against copy-synthesis labels. We expect it to lose on localization. If it matches the auditors, the specialized detectors are unnecessary.
Proposed Experiments
We train auditors on copy-synthesis outputs from LibriTTS and a subset of stress domains. Evaluation uses LibriTTS test-clean (in-distribution), VCTK and LJSpeech (unseen speakers), ESD (emotional extremes), and GTSinger and MUSDB18-HQ vocals (singing and music), with at least one of the last two held out from auditor training entirely.
Objective metrics are multi-resolution spectral error, PESQ, periodicity error and voiced/unvoiced F1 against references. We also report UTMOS, but because it is a baseline's selection criterion, we do not use it as the headline metric. Subjective quality is measured by MUSHRA-style listening tests on the out-of-distribution sets. Cost is reported as real-time factor per utterance, since repair adds compute and the latency concerns raised about multi-agent systems apply here.
The first decisive check is the localization measurement above. The second is the matched-compute comparison against frame-level untyped splicing, which isolates whether knowing the artifact type buys anything. The third is the verifier ablation: the rate at which accepted repairs increase any other artifact must drop sharply when the verifier is on. The fourth is auditor accuracy on the held-out domain, since a diagnosis that does not transfer makes the whole loop unreliable. Only if all four hold does the proposal's claim stand: splitting vocoder quality control into specialized, independently checked auditors fixes out-of-distribution artifacts that neither better architectures nor score-based selection remove.
References
- HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
- Avocodo: Generative Adversarial Network for Artifact-Free Vocoder
- BigVGAN: A Universal Neural Vocoder with Large-Scale Training
- AF-Vocoder: Artifact-Free Neural Vocoder with Global Artifact Filter
- Vocbench: A Neural Vocoder Benchmark for Speech Synthesis
- AgentSteerTTS: A Multi-Agent Closed-Loop Framework for Composite-Instruction Text-to-Speech
- Long-Video Audio Synthesis with Multi-Agent Collaboration
- Neural Homomorphic Vocoder
- Self-Collaboration Code Generation via ChatGPT
- ReChisel: Effective Automatic Chisel Code Generation by LLM with Reflection
- A survey on LLM-based multi-agent systems: workflow, infrastructure, and challenges
- UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022
- MACM: Utilizing a Multi-Agent System for Condition Mining in Solving Complex Mathematical Problems
- Beyond Lines and Circles: Unveiling the Geometric Reasoning Gap in Large Language Models