Human speech is the result of a physical partnership: the lungs and vocal folds produce a raw "source" signal (excitation), which is then shaped by the "filter" of the vocal tract (the throat, tongue, and lips). While modern neural text-to-speech (TTS) systems often model this process as a single end-to-end transformation, a growing body of research suggests that explicitly decoupling these two components leads to more controllable, efficient, and natural-sounding speech.
The Problem of Entanglement
In many standard neural models, acoustic features like pitch and timbre are deeply entangled. When a model learns to generate a Mel-spectrogram, it often struggles to modify the pitch without inadvertently changing the speaker's identity—a phenomenon often resulting in the "helium effect."
Bak et al. address this by proposing FastPitchFormant, which uses separate generators for formants (the vocal tract filter) and excitation (the sound source). By isolating the formant-related information from the pitch-related excitation, the authors demonstrate that they can manipulate prosody far more accurately than systems that model these features jointly. Similarly, Bae shows that leveraging internal gate-based mechanisms within neural vocoders can effectively separate pitch from timbre, allowing for pitch augmentation that preserves the speaker's unique vocal identity.
Neural Excitation Modeling
The "source" in speech is notoriously difficult to model mathematically because it contains complex, non-linear residuals. Recent work replaces traditional simplistic pulses with neural networks that generate a high-fidelity excitation signal.
- GlotNet: Juvela et al. introduced a speaker-independent vocoder that uses a WaveNet-style architecture to generate glottal excitation. This signal is then passed through a traditional all-pole filter derived from acoustic features.
- ExcitNet: Song et al. use a similar hybrid approach, arguing that focusing the neural network's capacity on the residual excitation—rather than the full waveform—reduces audible noise and handles phonetic transitions more smoothly.
- Physical Priors: Michelashvili and Wolf take this further in music synthesis by using a fundamental frequency-driven sine wave as a physical prior, constraining the model to follow a specific melodic contour while the network learns the non-linear transformations for timbre.
Disentanglement for Voice Conversion
Decoupling source and filter is particularly powerful for voice conversion, where the goal is to change the speaker's identity (the filter) while keeping the original prosody (the source).
The NANSY framework, proposed by Choi et al., mimics the human vocal system by combining a harmonic source and a vocal tract filter in the log-magnitude domain. This allows for zero-shot voice conversion and pitch shifting by manipulating the components independently. Chen et al. also utilize a neural source-filter decomposition to disentangle timbre from pitch and energy, achieving higher intelligibility in one-shot prosody transfer. Even in diffusion-based models, Choi et al. show that treating the glottal source and vocal tract filter as separate denoising targets improves the robustness of style transfer.
Efficiency through Hybrid Architectures
One of the most practical benefits of source-filter decomposition is computational efficiency. By offloading the spectral shaping to classic Digital Signal Processing (DSP) components, researchers can significantly reduce the workload of the neural network.
The Neural Homomorphic Vocoder, developed by Liu et al., uses a neural network to predict complex cepstrums, which then drive a classic DSP synthesis engine. The authors report that this approach requires over 100 times less computational power than state-of-the-art GAN-based vocoders. Similarly, He demonstrates a glottal neural vocoder that achieves WaveNet-level quality with nearly 300 times less processing power by focusing neural modeling solely on the excitation signal.
Grounding in Physical Reality
Finally, this decomposition allows models to incorporate non-acoustic data. In articulatory synthesis, Jaumard-Hakoun et al. show that predicting vocal tract parameters from ultrasound images of the tongue and video of the lips—while deriving the excitation from physical signals—improves the naturalness of the generated voice. In the realm of lip-to-speech synthesis, Pham et al. use a dual-pathway architecture to separate semantic content from acoustic prosody, demonstrating that the source-filter framework remains a robust inductive bias even when the input is entirely visual.
Key Foundational Work
Source-filter separation for articulation-to-speech synthesis (2004) — contributed 1 concept elements to this direction.
An Articulatory-Based Singing Voice Synthesis Using Tongue and Lips Imaging (2016) — contributed 1 concept elements to this direction.