When listening to synthetic speech from early text-to-speech (TTS) or voice conversion systems, a distinct signature often stands out: a mechanical, "buzzy," or muffled quality. Even when the underlying statistical models perfectly predict the intended melody and rhythm of a sentence, the final audio frequently sounds robotic.
This problem stems from the vocoder—the signal-processing engine responsible for reconstructing a continuous audio waveform from discrete acoustic parameters. To understand why vocoders introduce these artifacts, and how modern speech synthesis has overcome them, we must look at how traditional systems simplify the physics of human speech.
The Root of the Robotic Sound
Traditional parametric speech synthesis relies heavily on the source-filter model of speech production. This model conceptualizes human speech as a two-part system: an excitation source (the airflow from the lungs passing through the vocal cords) and an acoustic filter (the vocal tract, tongue, and lips that shape that airflow into specific phonemes).
To make this mathematically tractable, early parametric systems introduced severe simplifications. As noted by Heiga Zen et al. in their analysis of early hidden Markov model (HMM) synthesis systems, traditional vocoders relied on a simplistic binary excitation model. They represented voiced speech (like vowels) as a perfectly periodic pulse train and unvoiced speech (like the "s" sound) as aperiodic white noise.
[Periodic Pulse Train] ---\
+--> [Vocal Tract Filter] --> [Buzzy Synthetic Speech]
[Aperiodic White Noise] ---/
This binary assumption creates a fundamental mismatch with physical reality. Human vocal cords do not produce perfect mathematical impulses; instead, they produce complex, non-linear glottal flow waveforms with subtle variations in timing and texture. When a vocoder forces speech into a rigid pulse-or-noise framework, it discards the rich, organic details of the actual speech residual. This loss of acoustic transparency manifests directly as a metallic or buzzy artifact.
The Limits of Mathematical Simplification
The problem is compounded by how vocoders model the vocal tract filter. To simplify computation, conventional vocoders often enforce a minimum-phase assumption on the filters used to shape the spectrum. However, as explained by Zhijun Liu et al. in their work on the Neural Homomorphic Vocoder, real human speech exhibits mixed-phase characteristics. By ignoring these natural phase relationships, traditional vocoders fail to reproduce the natural temporal fine structures of speech, resulting in a flattened, artificial timbre.
Because these artifacts are baked into the signal representation itself, improving the underlying machine learning models cannot solve them. Alan Black demonstrated this limitation in CLUSTERGEN: a statistical parametric synthesizer using trajectory modeling, showing that even when a system successfully improves temporal trajectory modeling to make transitions smoother, the output retains a persistent "vocoded" quality. The bottleneck is not the prediction of the parameters, but the inability of the signal-processing engine to recover the intricate details lost during the initial analysis and decomposition phase.
Reconstructing the Excitation Signal
To eliminate buzziness without abandoning the efficiency of the source-filter framework, researchers first attempted to design richer, data-driven excitation models that could capture what a simple pulse train misses.
Instead of synthesizing an artificial excitation source from scratch, Zhengqi Wen et al. proposed a method in Pitch-Scaled Analysis based Residual Reconstruction for Speech Analysis and Synthesis that uses pitch-scaled analysis to extract and reconstruct the actual speech residual from real recordings, compressing it efficiently using Principal Component Analysis (PCA). By using a compressed version of a real human residual rather than a mathematical abstraction, the system significantly reduced buzziness, particularly in highly sensitive female voices. Similarly, June Sig Sung et al. utilized Waveform Interpolation (WI) to parameterize and smoothly transition the characteristic waveforms of the speech residual directly within statistical synthesis frameworks.
Other approaches sought to model the physical dynamics of the glottis more accurately. Manu Airaksinen et al. introduced GlottDNN - A Full-Band Glottal Vocoder for Statistical Parametric Speech Synthesis, which replaces impulse-like signals with a deep neural network trained to output realistic glottal flow waveforms.
From a pure signal-processing perspective, Hideki Kawahara et al. introduced Frequency Domain Velvet Noise (FVN). FVN functions as an all-pass filter impulse response, allowing researchers to generate excitation signals that span the entire spectrum from random noise to periodic pulses without the harsh phase regularities that human ears perceive as a metallic buzz.
Bypassing Decomposition via Waveform Modification
If deconstructing speech into highly simplified parametric representations introduces irreversible artifacts, why decompose it completely in the first place? A parallel line of research emerged around the idea of direct waveform modification.
Felipe Espic argued in Waveform Generation Based on Signal Reshaping for Statistical Parametric Speech Synthesis that instead of building speech from scratch using a source-filter model, synthesis engines could "reshape" existing, naturally recorded speech signals to match target acoustic parameters. Because this technique operates directly on a real waveform, it inherently preserves the organic micro-textures and phase relationships of human speech.
This strategy proved highly effective for voice conversion and articulatory speech modification. Patrick Lumban Tobing et al. and Kazuhiro Kobayashi et al. developed frameworks that alter speech by calculating a "spectrum differential"—a mathematical description of the exact acoustic difference between the source and target sounds. Instead of running the audio through a vocoder re-synthesis pipeline, they filter the original natural speech waveform directly with this differential.
[Natural Source Waveform] ---> [Apply Spectrum Differential Filter] ---> [High-Fidelity Target Speech]
(Preserves natural micro-prosody)
By modifying the existing waveform rather than rebuilding it from parameters, the authors successfully retained the natural micro-prosody and subtle spectral fluctuations of the source voice, bypassing vocoder degradation entirely.
The Neural Paradigm Shift
The ultimate resolution to vocoder-induced artifacts came from abandoning hand-engineered mathematical models altogether. Rather than trying to design a better set of rules to capture the physics of the vocal tract, deep generative architectures proved capable of learning the entire distribution of raw audio directly.
The breakthrough occurred when Aäron van den Oord et al. introduced WaveNet: A Generative Model for Raw Audio. WaveNet bypasses traditional vocoders by using a stack of dilated causal convolutions to generate raw audio waveforms one sample at a time. Because it operates directly on the raw signal, it learns to model the intricate, non-linear patterns, phase structures, and high-frequency noise components of human speech without making any simplifying assumptions about source-filter separation or minimum-phase constraints.
This neural waveform generation paradigm completely redefined how speech parameters are converted back into sound. In voice conversion, EngSiong Chng et al. demonstrated that a speaker-dependent WaveNet could be trained to map speaker-independent phonetic features (such as phonetic posteriorgrams) directly to raw waveforms. This eliminated the traditional "feature mismatch" problem where errors in fundamental frequency (F0) or spectral parameter estimation led to robotic distortions.
As shown during the The Voice Conversion Challenge 2018 by Jaime Lorenzo-Trueba et al., and further refined for low-resource scenarios by Li-Juan Liu et al., replacing signal-processing vocoders with neural waveform generators allowed synthetic speech to finally shed its characteristic buzziness, achieving a level of naturalness and clarity that closely mirrors natural human recordings.
Key Foundational Work
An overview of nitech HMM-based speech synthesis system for blizzard challenge 2005 (2005) — contributed 1 concept elements to this direction.
Unification Phonology: Another look at “synthesis-by-rule” (1990) — contributed 1 concept elements to this direction.