A long-term harmonic plus noise model for speech signals

Faten Ben Ali, Laurent Girin, Sonia Djaziri LarbiInterspeech 2011

TL;DR

This paper introduces a "long-term" version of the Harmonic Plus Noise Model (HNM) for speech signals. By modeling HNM parameters over extended time segments (hundreds of milliseconds) using smooth functions, they achieve significant data compression while maintaining high synthesis quality, paving the way for more efficient speech coding and manipulation.

What Problem Does This Paper Address?

Traditional speech analysis and synthesis systems typically operate on short-term (ST) frames, processing audio every 20 milliseconds. While this captures the rapid changes in speech, it leads to a large amount of redundant information, especially since speech parameters often evolve smoothly over longer periods. This redundancy is a bottleneck for applications like low-bitrate speech coding, storage, and signal manipulation (e.g., time stretching). Previous attempts to model these long-term correlations have focused on specific models like Linear Prediction Coding (LPC) or purely voiced speech. This paper aims to extend the benefits of long-term modeling to the more general Harmonic Plus Noise Model (HNM), which is crucial for handling mixed voiced and unvoiced speech segments.

Methods

The core innovation is the Long-Term Harmonic Plus Noise Model (LT-HNM). Instead of modeling parameters frame-by-frame, they define "long-term frames" which are continuous segments of speech (either voiced or unvoiced). Within these LT frames, the trajectories of HNM parameters are modeled using Discrete Cosine Models (DCM). DCMs are chosen for their ability to represent smooth signals efficiently with a small number of coefficients.

Specifically, they employ a two-band HNM, separating the frequency spectrum into a low-frequency harmonic band and a high-frequency noise band, with a "voicing cut-off frequency" (Fv) defining the boundary. The LT-HNM models the trajectories of:

  1. Fundamental Frequency (Fo): Modeled using DCM.
  2. Voicing Cut-off Frequency (Fv): Also modeled using DCM, with a specific criterion to ensure the modeled Fv stays close to actual harmonic frequencies.
  3. Spectral Amplitudes: A novel 2D-DCM modeling approach is used. This involves a first DCM applied in the frequency dimension to represent the spectral envelope (both harmonic and noise components) for each ST frame, and a second DCM applied along the time dimension to model the evolution of these spectral envelope coefficients over the LT frame. This 2D approach allows for a fixed-size representation of the spectral amplitudes, regardless of the number of harmonics or noise components.

The paper also details algorithms for optimizing the model orders (P for time-DCM, M for frequency-DCM) to balance data compression and modeling accuracy, using criteria based on modeling error.

Datasets

The experiments were conducted on a dataset of 24 speech sentences sampled at 16kHz, totaling 50 seconds of audio, spoken by 13 male and female speakers across 4 languages. This dataset was segmented into 291 LT sections, with a mean duration of 0.17 seconds. The paper does not introduce new benchmarks but evaluates its method against existing ST-HNM and a 1D-HNM approach.

Key Insights and Ideas

  • Long-term correlation is key for compression: Speech parameters don't change drastically from one frame to the next. Exploiting these smooth, long-term trends is a powerful way to reduce the amount of data needed to represent speech.
  • DCM is an effective tool for smooth trajectories: The Discrete Cosine Model is well-suited for capturing the gradual evolution of speech parameters like Fo, Fv, and spectral envelopes over extended periods.
  • 2D-DCM for spectral amplitudes is a significant step: Modeling spectral amplitudes in both frequency and time dimensions simultaneously allows for a compact and unified representation of mixed voiced/unvoiced speech, a challenge for simpler models.
  • Model order optimization is crucial: Automatically finding the right balance between compression and accuracy for the DCM orders (M and P) is essential for practical application.

Key Results

  • Significant Compression Gain: The LT-HNM achieves substantial compression. Compared to a standard ST-HNM, the LT-HNM reduces the coefficient rate by 91.5%. It also outperforms a 1D-HNM (which uses DCM for spectral envelope but ST processing for other parameters) by 46%.
  • Comparable Synthesis Quality: Despite the massive reduction in coefficients, the perceptual quality of synthesized speech using the LT-HNM is nearly the same as the 1D-HNM. Mean PESQ scores were 2.4 for LT-HNM versus 2.3 for 1D-HNM, while the ST-HNM scored 2.9. This indicates that the LT-HNM effectively captures the essential speech information.
  • Efficient Model Order Reduction: For Fo and Fv modeling, the LT model order is typically around 2, offering a significant reduction compared to the number of ST frames in an LT section.

Limitations and Future Work

  • Perceptual Criteria: The current model fitting relies on objective error metrics. The authors suggest future work should incorporate perceptual criteria for more refined LT model fitting.
  • Phase Modeling: The paper focuses on amplitude and frequency parameters. Future work could explore better LT modeling of phase trajectories.
  • Speech Transformation: The LT-HNM is seen as a promising tool for applications like time-stretching speech signals.
  • Low-Bitrate Coding: The ultimate goal is to design a low-bitrate speech coder based on the LT-HNM, potentially involving "LT-quantization" of its parameters.
  • "Birth and Death" of Harmonics: While managed, the independent modeling of Fo and Fv can lead to interruptions in harmonic trajectories, which is handled by a "birth and death" process. More seamless integration might be beneficial.

Summary updated

BibTeX
@inproceedings{ali2011long,
  author = {Faten Ben Ali and Laurent Girin and Sonia Djaziri Larbi},
  title = {{A long-term harmonic plus noise model for speech signals}},
  booktitle = {Interspeech},
  year = {2011},
}
Figure