A flexible spectral modification method based on temporal decomposition and Gaussian mixture model

Binh Phu Nguyen, Masato AkagiInterspeech 2007

TL;DR

This paper introduces a novel method for modifying speech spectra that addresses two key challenges: ensuring smooth transitions between modified audio frames and achieving more effective spectral alterations. By combining Temporal Decomposition (TD) for temporal smoothness and Gaussian Mixture Models (GMMs) for spectral flexibility, the authors demonstrate improved voice gender conversion and emotional speech synthesis.

What Problem Does This Paper Address?

Modifying speech spectra is crucial for various applications like voice conversion, speech enhancement, and emotional speech synthesis. However, existing methods often struggle with two main issues:

  1. Spectral Smoothness: Modifying speech spectra frame-by-frame can lead to abrupt changes, resulting in audible clicks and a degradation of overall speech quality. This is akin to trying to edit individual frames of a video without considering how they connect to the next.
  2. Spectral Modification Effectiveness: Traditional techniques often lack fine-grained control over specific spectral characteristics, such as formant frequencies and bandwidths. This makes it difficult to achieve precise and natural-sounding modifications. For instance, some methods might alter formant locations but struggle to independently control their width, leading to unnatural spectral envelopes.

The authors aim to overcome these limitations by developing a method that ensures smooth spectral evolution across time while offering flexible and precise control over spectral features.

Methods

The proposed method integrates two core techniques:

  • Temporal Decomposition (TD): This technique models the temporal evolution of speech spectra. Instead of modifying each frame independently, TD breaks down speech into "event targets" (representing desired spectral characteristics) and "event functions" (describing how these targets evolve over time). By modifying these event targets and functions, the method inherently ensures smoothness because the event functions dictate the transitions. The authors specifically use a Modified Restricted Temporal Decomposition (MRTD) algorithm, which enforces a "well-shapedness" property on event functions for more effective temporal modeling.
  • Gaussian Mixture Models (GMMs): GMMs are used to model the speech spectrum's envelope. Each Gaussian component in the GMM can be related to a formant's location, bandwidth, and amplitude. This provides a flexible representation that allows for independent control of these formant characteristics. The authors leverage GMM parameters (specifically, the mean components, which relate to formant locations) as input to the TD process.

The novel aspect lies in using the GMM parameters as input for TD. This allows the system to:

  1. Model the spectral envelope with GMMs, enabling flexible modifications.
  2. Use TD to ensure temporal smoothness by decomposing and modifying these GMM parameters (specifically, their means) according to formant scaling factors.

The process involves extracting GMM parameters, identifying spectral peaks, determining how much these peaks should shift (based on formant scaling factors), and then shifting the corresponding GMM mean parameters. Finally, the modified GMM parameters are used to reconstruct the modified spectral envelope.

Datasets

The paper mentions the following datasets:

  • ATR Japanese speech database: Used for experiments in voice gender conversion, specifically focusing on Japanese vowels spoken by male and female speakers.
  • Emotional speech utterances: Created from a neutral speech corpus, likely for the emotional speech synthesis experiments. The paper refers to [12] for details on how these were generated.

No new benchmarks were introduced; the focus was on evaluating the proposed method against existing ones.

Key Insights and Ideas

  • Temporal Smoothness via Event Functions: The core idea is that by modeling spectral evolution with event functions, the inherent shape of these functions dictates the smoothness of the modified speech, eliminating the need for frame-by-frame adjustments that can cause artifacts.
  • GMMs for Flexible Spectral Control: GMMs offer a powerful way to represent spectral envelopes, allowing for independent manipulation of formant-related characteristics, which is a significant improvement over methods that struggle with this.
  • Synergy of TD and GMMs: Combining TD with GMMs is the key innovation. TD provides the temporal structure and smoothness, while GMMs offer the fine-grained spectral control needed for effective modification. This addresses both major drawbacks of previous methods simultaneously.
  • Formant-Based Modification: The method directly links spectral modifications to formant scaling factors, making the process intuitive and aligned with how humans perceive speech characteristics.

Key Results

The proposed method was evaluated in two scenarios:

  • Voice Gender Conversion: The proposed method achieved superior or comparable results to existing methods (STRAIGHT+LSF and STRAIGHT+GMM) in terms of both correct gender identification and subjective mean opinion scores for speech quality. For instance, in M to F conversion, the proposed method achieved 93.8% correct identification and a mean opinion score of 3.19, compared to 83.3% and 2.73 for STRAIGHT+LSF.
  • Emotional Speech Synthesis: The proposed method also showed slightly better subjective listening results for emotional speech synthesis compared to a method based on formant modification in the LSF domain. The mean opinion score for the proposed method was 3.52, compared to 3.45 for the other method.

Qualitatively, the results suggest that the proposed method produces smoother and more natural-sounding modified speech.

Limitations and Future Work

  • GMM Parameter Smoothness: The paper acknowledges that the proposed method currently only ensures the smoothness of the mean components of GMM parameters. The authors believe that other GMM parameters (like standard deviations and mixture weights) could also be decomposed and modified using TD, which would further enhance the method's capabilities.
  • Exploring Other Parameters: Future work will investigate decomposing and modifying other GMM parameters by exploring relationships among parameters or using different types of event functions. This could lead to even more sophisticated spectral control.

Summary updated

BibTeX
@inproceedings{nguyen2007flexible,
  author = {Binh Phu Nguyen and Masato Akagi},
  title = {{A flexible spectral modification method based on temporal decomposition and Gaussian mixture model}},
  booktitle = {Interspeech},
  year = {2007},
}
Figure