Tying covariance matrices to reduce the footprint of HMM-based speech synthesis systems

Keiichiro Oura, Heiga Zen, Yoshihiko Nankaku, Akinobu Lee, Keiichi TokudaInterspeech 2009 · DOI

Abstract

This paper introduces a method to drastically reduce the memory footprint of HMM-based speech synthesis systems by tying all covariance matrices into a single global matrix. By leveraging the empirical observation that mean vectors are far more critical to speech quality than covariance matrices, the researchers were able to shrink model sizes by more than 50% while maintaining, or even slightly improving, the naturalness of the synthesized speech. This approach allows for higher-quality synthesis on memory-constrained embedded devices.

Introduction

In the field of speech synthesis, there is a constant trade-off between quality and "footprint"—the amount of memory a system consumes. While unit-selection synthesis offers high quality by using recorded waveforms, its massive database requirements make it unsuitable for embedded devices like mobile phones, GPS units, or game consoles.

HMM-based (Hidden Markov Model) synthesis has emerged as a popular alternative, storing statistical models rather than raw audio. However, even these models can be too bulky for highly constrained environments. Standard HMM systems (Figure 1) typically tie mean vectors and covariance matrices together within the same decision-tree structure. The authors identify this as a missed opportunity: since covariance matrices do not impact perceived quality as much as mean vectors do, they can be simplified to save space.

Overview of the HMM-based speech synthesis system, showing training and synthesis stages.

Methodology: Global Covariance Tying

The core of the proposed approach is a modified decision-tree-based context-clustering technique. In a standard HMM system, each leaf node in the decision tree contains both a mean vector and a covariance matrix. The authors propose decoupling these parameters to optimize storage.

As shown in Figure 5, the proposed method ties all covariance matrices across the entire system into a single, global diagonal covariance matrix. By removing the need to store thousands of individual covariance matrices, the system can reallocate its memory "budget" to grow much deeper and more detailed decision trees for the mean vectors.

The proposed global tying method, where all HMM states share a single global covariance matrix while maintaining individual mean vectors in the decision tree.

The researchers utilized the Minimum Description Length (MDL) criterion to automatically determine the optimal size of these trees. By modifying the likelihood calculation to account for the global covariance, the MDL-based clustering focuses entirely on refining the mean vectors to capture more contextual nuance.

Experimental Setup

The researchers evaluated their method using the ATR Japanese speech database B-set:

  • Training: 450 phonetically balanced sentences uttered by a male speaker (MHT).
  • Evaluation: 53 sentences.
  • Audio Specs: 16kHz sampling, using STRAIGHT mel-cepstral coefficients for spectral parameters and log $F_0$ for excitation.

Key Insights and Results

  • Perceptual Asymmetry: The human ear is significantly more sensitive to the accuracy of the mean (the average sound of a phoneme in context) than the variance (the statistical spread).
  • Quality Compensation: The saved memory allows for more leaf nodes for mean vectors. For instance, the proposed system used 1,311 leaf nodes for spectral means compared to the baseline's 808. This increased granularity compensates for the loss of detail in the covariances.
  • Significant Compression: A baseline system requiring 813 KB was outperformed by the proposed system at only 300 KB—a reduction of over 60%—while maintaining nearly identical speech quality. When comparing models of the same size, the proposed technique consistently delivered higher quality.

Conclusion and Future Work

Tying all covariance matrices is a highly efficient way to reduce the footprint of HMM-based synthesis without sacrificing naturalness. Future research may explore extending this technique to full covariance matrices or "semi-tied" models, where groups of states share matrices to find an even finer balance between quality and size. Additionally, further optimization of how global variance (GV) interacts with globally tied state covariances remains an open area for improvement.

Summary updated

BibTeX
@inproceedings{oura2009tying,
  author = {Keiichiro Oura and Heiga Zen and Yoshihiko Nankaku and Akinobu Lee and Keiichi Tokuda},
  title = {{Tying covariance matrices to reduce the footprint of HMM-based speech synthesis systems}},
  booktitle = {Interspeech},
  year = {2009},
  doi = {10.21437/interspeech.2009-143},
}
Figure