Diphone collection and synthesis

Kevin A. Lenzo, Alan W. BlackInterspeech 2000 · DOI

TL;DR

This paper details a streamlined and generalized process for collecting and synthesizing diphone corpora, crucial for building high-quality synthetic voices. They introduce a publically available database of ten diphone sets and a user-friendly tool, "pointyclicky," to simplify the recording and labeling process, making custom voice synthesis more accessible.

What Problem Does This Paper Address?

Building a synthetic voice, particularly one that sounds natural and human-like, is a complex undertaking. A common approach is diphone synthesis, which relies on collecting recordings of all possible two-phone transitions (diphones) in a language. Historically, the process of collecting and preparing these recordings was often ad-hoc, relying on fragmented scripts and anecdotal knowledge. This made it difficult for researchers and developers to reliably create new synthetic voices, especially for different speakers or languages. The authors aim to demystify and standardize this process, providing a robust framework and accessible tools for anyone to build their own diphone-based synthetic voices.

Methods

The core of their contribution lies in a generalized and streamlined methodology for diphone corpus collection. This involves:

  • Phone Set Selection: Utilizing a 44-phone inventory for American English, similar to those used in speech recognition systems.
  • Carrier Material Design: Generating nonsense carrier words to elicit diphones. This approach is favored over natural text to ensure more consistent pronunciation and avoid speaker assumptions.
  • Prompt Generation: Synthesizing prompts from the carrier material to guide the speaker during recording. This synthesized speech, delivered in a monotone "chant," helps ensure consistent pitch and duration, which are crucial for later modification.
  • Audio Collection Environment: Describing a practical setup using standard recording equipment (microphone, preamp, EGG, audio card) and software (Linux, FestVox tools). They emphasize the importance of a quiet environment and consistent recording levels.
  • "Pointyclicky" Tool: A novel Perl/Tk application designed to manage the recording session. It simplifies tasks like playing prompts, recording, labeling, and visualizing the progress of the recording session, significantly improving efficiency and reducing errors.
  • Segmentation/Alignment: Employing a dynamic time warping (DTW) based technique to align pre-generated, labeled synthesized prompts to the new recordings. This automates the initial labeling process.
  • Quality Control: Implementing visual inspection of alignments and running scripts to check durations and other features, followed by manual correction using tools like "emulabel."
  • Pitchmark Extraction and Normalization: Extracting pitchmarks from EGG signals (or waveforms if EGG is unavailable) and applying power normalization to account for variations in recording levels.
  • Diphone Indexing and Amalgamation: Creating a diphone index from the labeled data and combining it with other TTS components (text analysis, lexicons, prosodic models) to form a complete voice.

Datasets

The paper describes the creation of a new, publically available database of ten diphone sets from a single American English speaker (KAL). This database is released under an open license. They also mention using existing, hand-labeled databases for evaluating their automatic segmentation/alignment technique.

Key Insights and Ideas

  • Democratizing Voice Synthesis: The primary insight is that the complex process of diphone corpus collection can be generalized, documented, and tool-supported, making it accessible to a wider audience.
  • The Power of "Pointyclicky": The development of a user-friendly, interactive recording management tool is a significant practical contribution, directly addressing the tedium and error-proneness of manual session management.
  • Monotone Prompts for Control: Using synthesized, monotone prompts is a clever strategy to encourage consistent speaker delivery, which is essential for effective diphone extraction and modification.
  • Automated Alignment Efficiency: The DTW-based alignment technique significantly speeds up the labeling process, reducing it from days to minutes, while maintaining good quality for a first pass.
  • Generalizability: The described techniques have been successfully applied to other voices and languages, demonstrating the robustness of the framework.

Key Results

  • The paper details the collection of ten diphone sets for an American English speaker, released publicly.
  • The "pointyclicky" tool is presented as a significant improvement in recording session management, making the process more pleasant and manageable.
  • The automatic segmentation and alignment technique, when evaluated against hand-labeled data, showed promising results, even across different languages (e.g., English to Croatian/Nepali).
  • The authors claim that a complete US English diphone set can be recorded, labeled, and hand-checked in approximately a day, a significant speedup compared to previous methods.

Limitations and Future Work

  • Perfect Alignment is Elusive: While the automatic alignment is good, it's not perfect. Manual checking and boundary adjustments are still required for optimal performance.
  • Vowel-Vowel Transitions: The authors note that vowel-vowel transitions are particularly difficult to produce and are a common area for errors.
  • Difficult Phones: Phones like [AX] and [AH] are identified as being difficult to produce reliably, even for experienced speakers.
  • Future Work: The paper implies that further refinement of the tools and techniques will continue, with the goal of making voice building even more accessible and efficient. The availability of the data and tools on http://festvox.org suggests ongoing development and community engagement.

Summary updated

BibTeX
@inproceedings{lenzo2000diphone,
  author = {Kevin A. Lenzo and Alan W. Black},
  title = {{Diphone collection and synthesis}},
  booktitle = {Interspeech},
  year = {2000},
  doi = {10.21437/icslp.2000-537},
}

Loading related content...

Figure