Most text-to-speech systems work in two stages. An acoustic model turns text into a mel-spectrogram, a compact picture of which frequencies are loud at each moment. A neural vocoder then turns that picture into audio. The vocoder is usually trained on spectrograms computed from real recordings. At deployment time it only ever sees spectrograms predicted by the acoustic model, and these are blurrier and smoother than real ones. This train–test gap, called acoustic–vocoder feature mismatch, is a main source of buzzy, metallic synthetic speech. Separately, the push to run speech synthesis on watches, hearing aids, and phones has produced heavily compressed vocoders, such as spiking networks (networks that pass binary events instead of real numbers) and low-bit quantized networks. These compressed vocoders are almost always tested on clean, real spectrograms, which is exactly the input they will never see in a deployed cascade.
This note proposes that the two problems compound. Correcting blurred inputs is a job that needs numerical precision, and compression removes precision. We expect a compressed vocoder to degrade more on predicted spectrograms than the separate effects of compression and mismatch would suggest. If so, the fix is not uniform compression followed by fine-tuning. The fix is to find the few layers where the vocoder does its mismatch correction, keep only those at full precision, and compress everything else. The work first measures this interaction and then tests whether allocating precision by "mismatch sensitivity" beats uniform compression at equal energy.
Core Claim
Let $Q$ be a perceptual quality score such as UTMOS. Let $c \in \{0,1\}$ indicate whether the vocoder is compressed and $m \in \{0,1\}$ whether its input is predicted rather than real. Define the drop $\Delta(c,m) = Q(0,0) - Q(c,m)$. The quantity of interest is the interaction
$$I = \Delta(1,1) - \Delta(1,0) - \Delta(0,1),$$
which is zero if compression and mismatch hurt independently and positive if they compound. We make two claims. The first is that $I > 0$ for both spiking and 4-bit vocoders, even after each vocoder has been fine-tuned on predicted spectrograms. The second is that the lost quality is concentrated in a small number of layers whose internal activations differ most between real and predicted inputs. Keeping just those layers continuous or 8-bit, and making the rest spiking or 4-bit, should recover most of the drop at a small energy cost.
The file offers no evidence on this question. Spiking Vocos builds a spiking version of the Vocos vocoder. It uses an amplitude shortcut that re-injects continuous input amplitude and distils from an ANN teacher. It reaches 8.5 nJ versus 58.0 nJ for the ANN, at UTMOS 3.74 versus 3.82, but only on ground-truth LibriTTS spectrograms. Whether its shortcut passes upstream blur straight into the waveform is untested. Bunched LPCNet cuts vocoder compute by 54.2% with a 2.19× phone speedup and under 0.1 MOS loss inside a real cascade. However, it compresses by restructuring prediction, not by lowering arithmetic precision, and it does not study mismatch. CQNV is the closest in spirit. It shows that a vocoder with a dedicated full-precision input-processing module can absorb coarsely quantized codec parameters, reaching ViSQOL 3.123 at 1.1 kbps. We borrow its intuition that a small, precise input front-end can absorb upstream coarseness. CQNV quantizes the transmitted parameters, not the network, so it says nothing about where precision should live inside a compressed vocoder facing a neural acoustic model.
Research Program
The motivating evidence for mismatch comes from work showing that squared-error acoustic models produce over-smoothed spectrograms (Hosaka et al.). It also comes from work showing that GAN vocoders amplify that blur into audible flaws (Shi et al.) and from the observation that one-step GAN vocoders are particularly brittle under this shift (PeriodWave). The standard remedy is modeling-by-generation: train or fine-tune the vocoder on the acoustic model's predictions (MbG-ExcitNet, Hu et al.). We reuse this as a training ingredient for every system. The compression side supplies the methods. We use the spiking design of Spiking Vocos and ordinary quantization-aware training at 8 and 4 bits. The approximate computing survey acts as a constraint: it stresses that domain-specific error-tolerance models are missing and that uniform precision scaling is the default. Our allocation study is a domain-specific tolerance model for vocoders.
The program has three stages. In the measurement stage, we train a FastSpeech2-style acoustic model on LJSpeech and on LibriTTS. We generate predicted spectrograms with teacher-forced durations, so they stay time-aligned with the real waveforms and reference metrics like PESQ remain valid. We then fill the 2×2 grid of compressed/uncompressed and real/predicted inputs for three vocoder families: ANN Vocos with its spiking counterpart, HiFi-GAN with 8- and 4-bit versions, and each of these with and without modeling-by-generation fine-tuning. This yields $I$ with bootstrap confidence intervals over utterances.
In the localization stage, we take a full-precision vocoder fine-tuned on predicted spectrograms. For each block we compute a mismatch-sensitivity score: the normalized distance between its activations when fed real versus predicted spectrograms of the same utterance. We then check whether this score predicts the per-block quality damage from compressing that block alone under predicted inputs. We also run the same analysis on the Spiking Vocos amplitude shortcut to answer whether it leaks upstream blur.
In the allocation stage, we fix an energy budget equal to that of a uniformly spiking or 4-bit vocoder plus a small margin, for example 10%. Under that budget we keep the highest-scoring blocks continuous or 8-bit and compress the rest. We then train with modeling-by-generation data and distillation from the full-precision fine-tuned teacher. Energy is reported in the same per-operation accounting Spiking Vocos uses, alongside MAC/AC and bit-operation counts and measured CPU real-time factor. We state plainly that this is not a measurement on neuromorphic silicon.
Baselines
The naive reference point is the compressed vocoder exactly as published: Spiking Vocos, or 4-bit HiFi-GAN, trained only on real spectrograms and dropped into the cascade. On clean LibriTTS spectrograms Spiking Vocos reports UTMOS 3.74 and PESQ 3.45 against 3.82 and 3.65 for ANN Vocos (Spiking Vocos). No reported result found on Lacuna for any spiking or low-bit vocoder on predicted spectrograms, so we produce these numbers ourselves.
The strongest competitor is uniform compression plus modeling-by-generation fine-tuning. This means the same spiking or 4-bit vocoder fine-tuned on predicted spectrograms, with the same teacher distillation, the same data, and the same number of training steps as our method. Its energy is matched to ours by giving it a slightly wider network, which uses up the same margin we spend on full-precision blocks. This baseline is the obvious combination of the two literatures, and it is where the proposal most plausibly fails. The only published figure for modeling-by-generation in the file is MOS 4.57 versus 4.43 for the unrefined vocoder, on a Korean corpus with a different vocoder (MbG-ExcitNet). That figure is not directly comparable, so we rerun the approach. Beating it means a statistically significant gain in UTMOS and pitch error, plus a preference in a paired listening test, at equal energy.
Two ablations isolate the key ingredient. The first gives the same number of full-precision blocks to randomly chosen positions. The second gives them to blocks chosen by plain quantization sensitivity on real inputs. If sensitivity-guided allocation does no better than these, the "mismatch lives in specific layers" claim fails.
Two quality ceilings frame the results. One is the full-precision vocoder fine-tuned on predicted spectrograms. The other is 16-step PeriodWave, which the file credits with pitch error 15.042 versus 19.019 for BigVGAN on LibriTTS and with robustness to upstream errors at many times the inference cost. On LJSpeech we also report JETS as the end-to-end reference, which reports MOS 4.02 and MCD 7.16.
Why not just…?
Why not just fine-tune the compressed vocoder on predicted spectrograms? This is the real threat. If uniform compression plus modeling-by-generation already drives $I$ to zero, then precision allocation is unnecessary. The honest conclusion would be that the standard fix composes with compression, which is still an unreported and useful result. The measurement stage settles this directly, because $I$ is computed after this fine-tuning, and the matched-energy baseline above settles it for the method. We will change nothing about the conclusion to protect the idea.
Why not just quantize a mismatch-tuned full-precision vocoder after training? Post-training quantization is the cheapest path, but the file gives reasons to doubt it at low bit-widths. GAN vocoders amplify small input perturbations (Shi et al.). Even Bunched LPCNet needed a structural redesign of its output precision rather than plain truncation to avoid degradation (Bunched LPCNet). We include 8-bit and 4-bit post-training quantization in the grid. If 8-bit post-training quantization already has near-zero interaction, we restrict the claim to the 4-bit and spiking regimes.
Why not just distil a robust multi-step vocoder like PeriodWave into one step, or train end to end like JETS? Distillation targets inference steps, not arithmetic precision. A one-step distilled model still runs dense continuous multiplications. We do use a robust teacher, and we test the PeriodWave teacher as a variant of the distillation target. End-to-end training removes the mismatch, but it produces one monolithic network that cannot be split between a host processor and a low-power vocoding chip, and JETS was only shown on a single speaker. We report it as a reference rather than a competitor, because it answers a different deployment question.
Proposed Experiments
The decisive checks run on LJSpeech and LibriTTS, with VCTK held out for unseen speakers. The first check is whether $I$ is positive with a confidence interval excluding zero for spiking and 4-bit vocoders after modeling-by-generation fine-tuning. If it is not, the allocation stage is dropped and the paper reports the null result. The second is whether per-block mismatch sensitivity correlates with per-block compression damage under predicted inputs, measured by rank correlation across blocks and replicated across both vocoder families. The third is whether sensitivity-guided allocation beats both the uniform-compression-plus-fine-tuning baseline and the random-allocation ablation at equal counted energy. Quality is measured by UTMOS, PESQ on teacher-forced outputs, pitch error distance, and a crowd-sourced paired preference test. For the Spiking Vocos shortcut specifically, we measure high-frequency spectral flatness of outputs with the shortcut enabled and disabled under predicted inputs, to test whether it passes blur through.
A positive result would give designers of low-power speech hardware a concrete rule: which layers to leave precise, justified by where upstream error is corrected. A negative result would show that existing compressed vocoders can be deployed in cascades with ordinary fine-tuning. Either outcome fills a gap the cited compression papers leave open.
References
- Spiking Vocos: An Energy-Efficient Neural Vocoder
- Bunched LPCNet: Vocoder for Low-Cost Neural Text-To-Speech Systems
- CQNV: A Combination of Coarsely Quantized Bitstream and Neural Vocoder for Low Rate Speech Coding
- Voice Conversion Based on Trajectory Model Training of Neural Networks Considering Global Variance
- Can Knowledge of End-to-End Text-to-Speech Models Improve Neural Midi-to-Audio Synthesis Systems?
- PeriodWave: Multi-Period Flow Matching for High-Fidelity Waveform Generation
- Neural Text-to-Speech with a Modeling-by-Generation Excitation Vocoder
- The Ajmide Text-To-Speech System for Blizzard Challenge 2020
- Approximate Computing Survey, Part II: Application-Specific & Architectural Approximation Techniques and Applications
- JETS: Jointly Training FastSpeech2 and HiFi-GAN for End to End Text to Speech