The HMM synthesis algorithm of an embedded unified speech recognizer and synthesizer

Guntram Strecha, Matthias Wolff, Frank Duckhorn, Sören Wittenberg, Constanze TschöpeInterspeech 2009

TL;DR

This paper presents an embedded system that unifies speech recognition and synthesis using identical Hidden Markov Models (HMMs). They developed a novel, low-resource synthesis algorithm that allows for arbitrary voice customization from speaker-independent models, significantly improving speech quality for embedded applications.

What Problem Does This Paper Address?

Developing speech recognition and synthesis systems for embedded devices presents a significant challenge due to limited computational power and memory. Traditionally, these systems require separate, often complex models. Furthermore, achieving speaker-specific synthesis from generic, speaker-independent models typically involves computationally intensive adaptation techniques that are unsuitable for resource-constrained environments. This paper tackles the problem of creating a unified, efficient system that can both understand and speak, with the ability to adapt its voice without requiring extensive hardware resources.

Methods

The core of the system is the use of identical speaker-independent HMMs for both recognition and synthesis. For synthesis, the authors propose a novel algorithm designed for embedded systems. Instead of recalculating speech parameters from scratch, they utilize pre-calculated Gaussian sequences (GS) for a limited set of speech units (morphemes and words), stored in a database. This approach significantly reduces computation.

To address the speaker-independent nature of the HMMs, they employ a linear transformation of speech parameters using a speaker-dependent conversion matrix. This matrix is learned by minimizing the mean square error between synthesized and natural speech parameters. The synthesis process involves retrieving GS indices from the unit database, transforming the parameters using the learned matrix, and then generating the speech signal using a mel-cepstrum synthesis filter. This filter is realized with an outer and inner filter structure to approximate the necessary functions.

Datasets

The paper describes the generation of a unit database using speech signals from three tested speakers (Speaker A, C, and M). These signals were used to train the HMMs and extract speech parameters. The evaluation involved listening tests using synthesized speech generated from these unit databases and conversion matrices.

Key Insights and Ideas

  • Unified HMMs: Using the same HMMs for both recognition and synthesis is a key to reducing memory footprint and computational complexity in embedded systems.
  • Pre-calculated GS for Synthesis: Storing pre-computed Gaussian sequences for speech units, rather than recalculating them, is a clever way to drastically reduce synthesis computation.
  • Linear Voice Conversion: A simple linear transformation with a speaker-dependent matrix is surprisingly effective for adapting the voice from speaker-independent models, especially for a limited set of synthesis units.
  • Embedded System Focus: The entire design is driven by the constraints of embedded systems, prioritizing efficiency in both computation and memory.

Key Results

The paper reports significant improvements in speech quality through their proposed synthesis method.

  • DMOS Listening Test: Synthesized speech converted to a target voice showed a significant improvement compared to unconverted speech. For example, when using units of Speaker A and converting to Voice A, the DMOS score increased from 1.61 to 2.68.
  • MOS Listening Test: The overall quality of synthesized speech, when using combinations of unit databases and conversion matrices, was substantially higher than unconverted signals. For instance, converting units of Speaker A to Voice A resulted in a MOS score of 2.67, compared to a much lower score for unconverted speech.
  • Efficiency: The unit database size is manageable (around 18 kByte per voice), and the conversion matrix size is constant (1860 values), fitting well within embedded system constraints.

Limitations and Future Work

  • Limited Synthesis Units: The effectiveness of the voice conversion relies on a limited set of synthesis units. The paper mentions 39 units for numerical values and status information. Expanding this set might require more sophisticated adaptation techniques.
  • Global Prosody Contour: The current approach might lead to a monotone prosody due to the lack of a global prosody contour. The authors mention storing variants of units dependent on their position in a sentence to mitigate this.
  • Future Work: The authors aim to reduce the hardware to the size and power consumption of a USB-stick, suggesting further miniaturization and optimization are planned. The system is envisioned as a speech interface for sound and vibration analyzers.

Summary updated

BibTeX
@inproceedings{strecha2009hmm,
  author = {Guntram Strecha and Matthias Wolff and Frank Duckhorn and Sören Wittenberg and Constanze Tschöpe},
  title = {{The HMM synthesis algorithm of an embedded unified speech recognizer and synthesizer}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure