Hybrid Sinusoidal and Stochastic Speech Excitation Modeling

40 concepts32 papers1 proposal

Human speech is an acoustic compromise between order and chaos. When we pronounce a voiced vowel, our vocal cords vibrate periodically, creating a highly structured harmonic signal. When we pronounce an unvoiced consonant like "s", air rushes through a constriction, creating turbulent, stochastic noise.

For decades, speech processing systems struggled to model this duality. Early speech synthesizers and codecs relied on a rigid binary assumption: a given frame of speech was either entirely periodic (modeled as a train of impulses) or entirely stochastic (modeled as white noise). This binary switch introduced a distinct, artificial "buzziness" or robotic quality because real human speech is rarely purely one or the other. Even during a clear vowel, there is breathy turbulence; during a voiced fricative like "z", periodic vibration and noise coexist simultaneously.

To bridge this gap, researchers developed the hybrid deterministic-stochastic framework—often realized as the Harmonic plus Noise Model (HNM) or sinusoidal speech model. By decomposing speech into a simultaneous mixture of a deterministic component (a sum of harmonically related sinusoids) and a stochastic component (filtered noise), this approach provides a mathematically elegant and physically accurate representation of speech production.

The Dual Nature of Speech Production

The physical intuition behind hybrid modeling traces back to early simulations of the human vocal tract. Paul Mermelstein, in Computer Simulation of Articulatory Activity in Speech Production, pioneered a dual-source excitation model that integrated a periodic pulse generator for glottal vibration alongside a white-noise generator for turbulence. This established the foundational idea that speech synthesis requires a continuous interplay between periodic and aperiodic airflow.

Mathematically, a hybrid model represents the speech signal $s(t)$ as:

$$s(t) = d(t) + n(t)$$

where $d(t)$ is the deterministic component and $n(t)$ is the stochastic component.

The deterministic component captures the periodic structure of voiced speech by modeling it as a summation of sinusoids at integer multiples of a fundamental frequency ($F_0$):

$$d(t) = \sum_{k=1}^{K} A_k(t) \cos\left(2\pi k F_0 t + \phi_k(t)\right)$$

Here, $A_k(t)$ and $\phi_k(t)$ represent the time-varying amplitudes and phases of each harmonic. As demonstrated by Meysam Asgari et al. in Improving the accuracy and the robustness of harmonic model for pitch estimation, this maximum-likelihood framework allows systems to analytically isolate periodic speech structures from background noise.

Tracking Phase and Eliminating Buzziness

One of the greatest challenges in sinusoidal modeling is maintaining phase continuity across successive short-time Fourier transform (STFT) frames. If the phases $\phi_k(t)$ of the sinusoids are misaligned between frames, the reconstructed audio suffers from phase cancellation and harsh artifacts.

To resolve this, Timo Gerkmann, in Least squares estimate of the initial phases in STFT based speech enhancement, showed that modeling voiced speech as a deterministic sum of sinusoids provides a structured framework to analytically predict how phase should evolve over time. Instead of relying on noisy phase measurements, a least-squares estimate can anchor the phase evolution based on the fundamental frequency.

On the synthesis side, combining these components smoothly is essential to eliminating the "buzziness" of older vocoders. Ranniery Maia et al., in A trainable excitation model for HMM-based speech synthesis, replaced the binary switch with a continuous architecture: a voiced component generated from a pulse train passed through a finite impulse response (FIR) filter, summed with an unvoiced component created by passing white noise through an all-pole filter.

   [ Fundamental Frequency F0 ] --------> [ Sinusoidal / Pulse Generator ] ---> [ Voiced Filter ] ----+
                                                                                                      |
                                                                                                      v
                                                                                                ( Summation ) ---> [ Mixed Excitation ]
                                                                                                      ^
                                                                                                      |
   [ Stochastic Noise Source ] ---------> [ Noise Modulator / Generator ] ----> [ Unvoiced Filter ] --+

This concept was further refined by Yannis Agiomyrgiannakis et al. in Towards flexible speech coding for speech synthesis: an LF + modulated noise vocoder, who introduced a modulated noise model where the stochastic residual is dynamically shaped by the deterministic glottal flow waveform. This ensures that the noise component mimics the natural amplitude modulation caused by the opening and closing of the vocal cords.

Similarly, Chang-Heon Lee et al., in A Hybrid Quasi-Harmonic/CELP Wideband Speech Coding Scheme for Unit Selection TTS Synthesis, paired an adaptive quasi-harmonic model (aQHM) with an innovative codebook to achieve high-fidelity compression of acoustic inventories, while Eunwoo Song et al., in Deep neural network-based statistical parametric speech synthesis system using improved time-frequency trajectory excitation model, leveraged this decomposition to independently model the slowly evolving periodic trajectories and rapidly changing stochastic trajectories of the speech waveform.

Inductive Priors for Neural Speech Generation

With the advent of deep learning, hybrid sinusoidal modeling has transitioned from a pure synthesis tool into a powerful inductive bias for neural networks. Deep generative models often struggle to reconstruct sharp, high-frequency harmonic lines or long, sustained musical notes, leading to glitching and phase artifacts. Explicitly feeding a sinusoidal skeleton into a neural network alleviates this burden.

In the domain of singing voice synthesis, where pitch stability is paramount, Feiyang Chen et al. developed SingGAN: Generative Adversarial Network For High-Fidelity Singing Voice Generation. The authors introduced a pitch-conditioned source excitation module that generates a sine-based signal for voiced segments and Gaussian noise for unvoiced segments, providing a stable harmonic backbone that guides the generator.

For highly expressive speech, Yuma Shirahata, in Period VITS: Variational Inference with Explicit Pitch Modeling for End-to-end Emotional Speech Synthesis, integrated an explicit periodicity generator that combines sample-level sinusoidal waves with stochastic noise and voicing flags. This allows the model to accurately capture the wide prosodic swings of emotional speech without losing tracking stability.

Furthermore, Qinghua Wu et al., in Effective and Differentiated Use of Control Information for Multi-speaker Speech Synthesis, demonstrated that calculating explicit multiples of $F_0$ to construct a synthetic "excitation spectrogram" provides a structural blueprint that helps neural decoders predict final mel-spectrograms with significantly lower reconstruction error. This structural guidance is mirrored in work by Yamato Ohtani et al. in HMM-based speech synthesis using sub-band basis spectrum model, which maps multi-pulse trains derived from $F_0$ and phase alongside banded aperiodic noise to reconstruct fine spectral details.

Exploiting Duality for Robust Speech Analysis

The hybrid model is equally powerful for analysis, denoising, and source separation. Ambient environmental noise is typically unstructured and stochastic. Because voiced speech conforms strictly to a rigid harmonic grid, fitting a sinusoidal model acts as a highly effective non-linear filter.

Hiromasa Fujihara et al., in Speaker identification under noisy environments by using harmonic structure extraction and reliable frame weighting, leveraged this property for robust speaker identification. By extracting the fundamental harmonic structure and reconstructing a clean, sinusoidal-only version of the speech, the authors filtered out non-stationary environmental noise without needing to explicitly model or subtract the noise itself. Xiang Zhang et al., in Speaker recognition using the resynthesized speech via spectrum modeling, utilized a similar strategy, decomposing speech into its deterministic parameters to isolate voiced segments and strip away cross-channel distortions.

For speech enhancement in low signal-to-noise ratio (SNR) environments, deep neural networks can be trained to predict these components explicitly. Zhiheng Ouyang et al., in A Deep Neural Network Based Harmonic Noise Model for Speech Enhancement, designed a network that targets the estimation of harmonic and residual amplitudes separately, ensuring that the underlying voice structure is preserved even when submerged in heavy noise.

To prevent the phase errors common to complex spectral addition during enhancement, Zhongshu Hou, in SNR-Progressive Model with Harmonic Compensation for Low-SNR Speech Enhancement, developed a progressive mechanism that estimates a speaker's pitch from a partially cleaned intermediate signal, extracts the voice structures via a comb filter, and blends them back using magnitude-based masking.

This iterative refinement is a common theme in parameter estimation. Alexander Pavlovets et al., in Robust HNR-based closed-loop pitch and harmonic parameters estimation, utilized a closed-loop analysis-by-synthesis approach that iteratively refines pitch and harmonic parameters based on the Harmonic-to-Noise Ratio (HNR), ensuring robust tracking in adverse conditions.

Multi-Source Separation and Complex Domain Tracking

When multiple people speak simultaneously, or when speech is recorded in echoey, convolutive environments, the hybrid model provides a mathematical anchor for source separation. Rasmus Kongsgaard Olsson et al., in A Harmonic Excitation State-Space Approach to Blind Separation of Speech, formulated a generative state-space framework where overlapping speech signals are modeled as combinations of harmonic tones and colored noise. By framing the problem this way, the authors demonstrated that it is possible to perform blind source separation and pitch tracking simultaneously using Kalman smoothing.

Ultimately, treating speech as a hybrid signal changes how we process audio mathematically. Rather than operating solely on magnitude spectra, modern algorithms model this duality directly within the complex Short-Time Fourier Transform (STFT) domain.

Matthew McCallum et al., in Stochastic-deterministic signal modelling for the tracking of pitch in noise and speech mixtures using factorial HMMs, and in their companion work on Joint stochastic-deterministic wiener filtering with recursive Bayesian estimation of deterministic speech, proved that directly modeling the predictable (deterministic) and unpredictable (stochastic) components in the complex STFT domain yields superior tracking and filtering performance. This approach focuses on the underlying physical structure of the signal, making it highly robust to amplitude variations and ambient noise compared to methods that treat speech as a uniform, monolithic wave.

Key Foundational Work

A Harmonic Excitation State-Space Approach to Blind Separation of Speech (2004) — contributed 1 concept elements to this direction.
Application of voice conversion to hearing-impaired Mandarin speech enhancement (2004) — contributed 1 concept elements to this direction.
Computer Simulation of Articulatory Activity in Speech Production (1969) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_sinusoidal_speech_representation_6790971f,
  title = {sinusoidal speech representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sinusoidal-speech-representation/rdr_6790971fe3824f8ea075356e340a5421},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_sinusoidal_speech_representation_6790971f_v1,
  title = {sinusoidal speech representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sinusoidal-speech-representation/rdr_6790971fe3824f8ea075356e340a5421/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake