A general-purpose 32 ms prosodic vector for hidden Markov modeling

Kornel Laskowski, Mattias Heldner, Jens EdlundInterspeech 2009

TL;DR

This paper introduces a new, efficient prosodic feature called the Fundamental Frequency Variation (FFV) spectrum, designed to capture instantaneous intonation patterns. By optimizing its representation and integrating it with standard acoustic modeling techniques, the authors achieve significant improvements in dialog act recognition tasks, demonstrating its potential for broad application in speech processing.

What Problem Does This Paper Address?

Modeling prosody, particularly intonation, has been a long-standing challenge in speech technology. Traditional methods often struggle with the frame-level discontinuities of pitch and the need for extensive normalization, making them difficult to apply in real-time systems. This paper aims to address this by proposing a novel, instantaneous representation of intonation that is robust, computationally efficient, and suitable for standard modeling paradigms like Hidden Markov Models (HMMs).

Methods

The core of this work is the Fundamental Frequency Variation (FFV) spectrum. This is a 7-element feature vector that characterizes within-frame changes in fundamental frequency without explicitly estimating the fundamental frequency itself. The authors detail a process involving:

  • Signal Pre-processing: Applying pre-emphasis and framing the signal into 32 ms overlapping windows with an 8 ms step.
  • Spectrum Computation: Calculating two frequency spectra (left and right halves of each frame) and then dilating them in frequency.
  • Alignment Measure: Computing a measure of harmonic train alignment for various dilation factors.
  • Filterbank Application: Oversampling the alignment measure and passing it through a filterbank motivated by psychoacoustic studies, resulting in the 7-element FFV vector.

Key methodological contributions include:

  • Feature Space Rotation: Investigating the benefits of Z-transformations (mean subtraction and variance scaling) and Principal Component Analysis (PCA) for improving classification.
  • Modified Filterbank: Developing a new filterbank structure based on a quadratic approximation of the response locus, aiming for better robustness.
  • Feature Combination: Exploring the synergy of FFV features with other prosodic correlates like loudness, voicing, and speaking rate (e.g., MFCCs).
  • Runtime Improvements: Optimizing the computation of extremity filters within the FFV spectrum to significantly reduce processing time without sacrificing accuracy.
  • Standard GMM Modeling: Demonstrating the applicability of standard HMMs with Gaussian Mixture Models (GMMs) for modeling FFV features, exploring variations in model complexity.

Datasets

The research utilizes the ICSI Meeting Corpus and its associated MRDA dialog act annotations. This corpus is described as the largest publicly available collection of naturally occurring multiparty conversations, comprising over 66 hours of meeting time. The data is divided into:

  • TRAINSET: 51 meetings for training.
  • DEVSET: 11 meetings for development and parameter tuning.
  • EVALSET: 11 meetings for unseen testing.

The experiments focus on four binary classification tasks related to dialog acts (specifically, floor holders and holds), with data drawn separately for each task. The tasks are balanced to ensure an equal number of instances for both classes, with the number of exemplars capped by the minority class in each task.

Key Insights and Ideas

  • FFV as a General Prosodic Feature: The paper strongly advocates for the FFV spectrum as a versatile and effective representation of instantaneous intonation, suitable for a wide range of speech processing tasks beyond its initial applications.
  • Complementarity of Features: FFV features are shown to be complementary to other acoustic measures, particularly those related to loudness, voicing, and speaking rate, leading to improved performance when combined.
  • Efficiency Gains: Significant reductions in computational complexity (a factor of five) are achieved through clever optimization of the FFV feature extraction process, making it more practical for real-time applications.
  • Robustness through Filterbank Design: A modified filterbank structure, derived from psychoacoustic principles, demonstrates improved performance, suggesting it better approximates human perception of intonation.
  • Standard Models Work Well: The FFV features are shown to be effectively modeled by standard HMMs and GMMs, indicating their compatibility with existing speech processing pipelines.

Key Results

The paper reports significant improvements across four dialog act recognition tasks:

  • Overall Error Reduction: Agglomerated error reduction on a development set was 11.1% absolute (39.0% relative), and on an unseen test set, it was 10.3% absolute (35.4% relative).
  • Runtime Improvement: Computation time was reduced by a factor of five with minimal impact on accuracy.
  • Feature Combination: Combining FFV with auxiliary features (loudness, voicing, speaking rate) and applying PCA led to improvements of 0.7% and 0.9% in classification and ROC discrimination errors, respectively, relative to FFV alone.
  • Model Complexity: Increasing the complexity of HMMs and GMMs (e.g., 8 states, 3 Gaussians) yielded further reductions in classification and ROC discrimination errors by 2.8% and 3.5%, respectively.

Limitations and Future Work

  • Computational Cost: While runtime improvements were significant, the paper notes that FFV computation is still "approximately realtime, making it relatively expensive" compared to some other features.
  • Data Size for Specific Tasks: For tasks 1B and 2B, which involve mid-talkspurt segmentation, the amount of training material was relatively small, potentially limiting the impact of some improvements on EVALSET data for these specific tasks.
  • Model Optimization: The authors acknowledge that the explored HMM/GMM parameters were not optimized for a global optimum but rather for demonstration purposes. Future work could involve more extensive parameter tuning on larger corpora.
  • Potential Applications: The authors suggest that the FFV representation is directly deployable in various speech processing applications, including discrimination for tonal languages, and for implementing competing dialog act productions.

Summary updated

BibTeX
@inproceedings{laskowski2009general,
  author = {Kornel Laskowski and Mattias Heldner and Jens Edlund},
  title = {{A general-purpose 32 ms prosodic vector for hidden Markov modeling}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure