Deviation measure of waveform symmetry and its application to high-speed and temporally-fine F0 extraction for vocal sound texture manipulation

Hideki Kawahara, Masanori Morise, Ryuichi Nisimura, Toshio IrinoInterspeech 2012

TL;DR

This paper introduces a novel, high-speed algorithm for extracting fundamental frequency (F0) from speech by measuring waveform symmetry. This method allows for unprecedented temporal resolution, enabling the analysis and manipulation of subtle vocal "textures" that contribute to expressiveness in singing and speech.

What Problem Does This Paper Address?

Analyzing and synthesizing expressive speech, particularly singing and dramatic vocal performances, is challenging because these vocalizations often contain rapid, irregular fluctuations in pitch (F0) and spectral envelope. Traditional F0 extraction methods, designed for more regular speech, struggle to capture these fine temporal details. This limitation hinders our ability to understand what makes expressive voices sound the way they do and to manipulate these characteristics for creative purposes. The authors aim to develop a faster and more temporally precise F0 extractor that can capture these subtle, expressive nuances.

Methods

The core of the proposed method is a "deviation measure of waveform symmetry." Instead of directly estimating F0, the algorithm identifies the lowest prominent sinusoidal component in the speech signal by analyzing how closely the waveform's shape adheres to a perfect sinusoid. This is achieved by:

  • Low-pass Filtering: Using a set of low-pass filters to isolate the fundamental component. The paper argues that low-pass filters offer better temporal resolution than traditional band-pass filters for this task.
  • Symmetry Measurement: Defining a measure that quantifies the deviation of the waveform from perfect symmetry. This deviation is related to how well the waveform can be approximated by a pure sinusoid.
  • Periodicity Index: A formula, $\eta_\epsilon$, is derived from this symmetry deviation measure. A value close to 1 indicates high periodicity (sinusoidal behavior), while lower values indicate deviation.
  • Adaptive Filter Selection: The algorithm selects the optimal low-pass filter based on this periodicity index, effectively choosing the filter that best captures the fundamental component for a given segment of the signal.
  • Refinement Procedure: For periodic signals with harmonic components, a two-stage refinement process uses the instantaneous frequencies of higher harmonics to improve the initial F0 estimate. This is crucial for increasing accuracy.

The authors emphasize that this method is a reformulation and improvement over their previous work, offering increased speed and accuracy.

Datasets

The paper primarily uses:

  • Artificial Test Signals: Periodic pulse trains with varying Signal-to-Noise Ratios (SNRs) and frequency-modulated harmonic sinusoids. These are used to evaluate the algorithm's robustness and temporal tracking capabilities.
  • Natural Speech: Japanese vowel sequences and singing samples from a male pop singer. These are used to demonstrate the algorithm's performance on real-world vocalizations and to analyze expressive vocal textures.

No new benchmarks are introduced; instead, the proposed method is compared against existing F0 extractors like XSX, YIN, and SWIPE.

Key Insights and Ideas

  • Symmetry as a Proxy for Periodicity: The paper cleverly reframes F0 extraction not as directly measuring frequency, but as measuring how "sinusoidal" a waveform segment is. This is a more robust approach for handling irregularities.
  • Temporal Resolution is Key to Expressiveness: The authors found that expressive voices, especially singing, exhibit rapid temporal modulations in both F0 and spectral envelope. Capturing these fine details is essential for understanding and manipulating vocal expressiveness.
  • Low-Pass Filtering for Finer Temporal Detail: The argument for using low-pass filters over band-pass filters for isolating the fundamental component is a novel perspective that directly leads to improved temporal resolution.
  • "Musical Growl" Texture: The analysis of singing samples revealed that expressive performances, particularly those described as "musical growl," possess a distinct temporal fine texture in their spectral envelope that is absent in less expressive versions.

Key Results

  • Speed and Accuracy: The proposed method is significantly faster than previous approaches, processing 1 second of speech in 0.2 seconds without refinement and 0.5 seconds with refinement (on a specific hardware configuration). It also demonstrates improved accuracy, particularly at lower SNRs.
  • Tracking Fast Modulations: The algorithm successfully tracks very fast frequency modulations (up to 64 Hz) in artificial signals, outperforming conventional F0 detectors.
  • Analysis of Expressive Voices: The method successfully identified and visualized fine temporal structures in expressive singing that are not captured by other F0 extractors.
  • Perceptual Impact: Preliminary tests suggest that smoothing these fine temporal structures can selectively reduce the "liveliness" of an expressive voice while preserving vocal effort and register, transforming a "musical growl" into a "strong and loud" voice.

Limitations and Future Work

  • Noise Tolerance: The paper notes that the proposed method without refinement is not inherently noise-tolerant. The refinement procedure helps, but further improvements in noise robustness might be needed.
  • Computational Cost of Refinement: While faster than previous methods, the refinement stage still adds computational overhead.
  • Future Work: The authors plan to conduct more systematic investigations into the perceptual effects of these fine temporal structures and integrate the proposed method with other acoustic event-related procedures within the STRAIGHT framework. They also suggest exploring the relationship between their method and other "True-Envelope based" approaches.

Summary updated

BibTeX
@inproceedings{kawahara2012deviation,
  author = {Hideki Kawahara and Masanori Morise and Ryuichi Nisimura and Toshio Irino},
  title = {{Deviation measure of waveform symmetry and its application to high-speed and temporally-fine F0 extraction for vocal sound texture manipulation}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure