Objective Intelligibility Assessment of Text-to-Speech System using Template Constrained Generalized Posterior Probability

Linfang Wang, Lijuan Wang, Yan Teng, Zhe Geng, Frank K. SoongInterspeech 2012 · OpenReview

TL;DR

This paper introduces a novel objective method for evaluating the intelligibility of Text-to-Speech (TTS) systems. By leveraging a technique called Template Constrained Generalized Posterior Probability (TCGPP), the authors can automatically assess how understandable synthesized speech is, achieving high correlation with human judgments and outperforming existing automatic methods. This offers a much-needed scalable and cost-effective alternative to laborious human listening tests for TTS development.

What Problem Does This Paper Address?

Evaluating the intelligibility of synthesized speech is crucial for developing high-quality Text-to-Speech (TTS) systems. Traditionally, this is done through subjective listening tests where human experts assess how well synthesized speech matches the intended message. However, these tests are expensive, time-consuming, prone to inconsistencies, and difficult to scale for comprehensive evaluation. Existing objective measures, like the Articulation Index (AI) and Speech Transmission Index (STI), are often limited to linear systems and not well-suited for modern TTS. While some speech recognition-based approaches have shown promise, they haven't been extensively validated for TTS intelligibility. The authors aim to bridge this gap by proposing a robust, automatic objective measure that can accurately reflect human perception of TTS intelligibility.

Methods

The core of the proposed method is the Template Constrained Generalized Posterior Probability (TCGPP). TCGPP is an extension of Generalized Posterior Probability (GPP) that uses "templates" to constrain the search space for matching speech hypotheses against a given text. These templates can be quite flexible, allowing for variations in context and even partial matches.

Here's how it's applied to TTS intelligibility:

  1. Synthesize Speech: The TTS system under evaluation generates audio from a given text script.
  2. Phone Graph Decoding: An Automatic Speech Recognition (ASR) system processes the synthesized audio to generate a "phone graph," which represents possible phone sequences and their probabilities.
  3. Forced Alignment: This step aligns the input text's phonetic transcription with the acoustic signals to determine precise start and end times for each phoneme.
  4. Confusable Phone Pairs: The acoustic model is used to identify pairs of phonemes that are acoustically similar (confusable), using measures like Kullback-Leibler Divergence (KLD).
  5. Template Construction & TCGPP Calculation: For each "focused" phoneme in the input transcription, multiple templates are constructed. These templates incorporate the focused phoneme, its surrounding context (left and right), and potentially confusable phonemes. The templates can vary in their strictness (e.g., requiring exact matches vs. allowing partial matches or wildcards). TCGPP is then calculated for each template, essentially measuring the confidence that the synthesized speech segment corresponding to the template accurately represents the intended phonemes and context.
  6. Overall Intelligibility Score: Multiple TCGPP values, calculated at different "granularity levels" (phone, syllable, word), are combined into a single overall objective intelligibility score (OIS). This score is designed to reflect the overall understandability of the synthesized speech.

The novelty lies in the flexible template construction and the application of TCGPP, which was previously used for speech transcription verification, to the specific problem of TTS intelligibility assessment at a fine-grained level.

Datasets

  • Internal Microsoft TTS Systems: Used for initial experiments comparing different TTS backend technologies (unit selection vs. HMM-based). A test set of 330 sentences was used.
  • Blizzard Challenge 2010 EH1 Task: This is the primary benchmark dataset. It includes synthesized speech from 17 different TTS systems participating in the challenge, evaluated on 100 sentences from the SUS (Semantically Unpredictable Sentences) session. This dataset is used to validate the proposed method against human intelligibility scores and ASR performance.

Key Insights and Ideas

  • Fine-Grained Error Detection: TCGPP's ability to use flexible templates allows it to pinpoint subtle errors in synthesized speech, such as weak reproductions of phonemes or slight misalignments, which are common in modern TTS systems.
  • Multi-Granularity Assessment: By calculating TCGPP at different levels (phone, syllable, word) and combining them, the method provides a comprehensive view of intelligibility, moving beyond simple phoneme accuracy.
  • Template Flexibility is Key: The ability to construct various templates, from strict to loose, is crucial. This allows the system to adapt to different types of potential errors and measure confidence at varying levels of precision, mimicking how humans might perceive different kinds of intelligibility issues.
  • TCGPP as a Robust Confidence Measure: The paper demonstrates that TCGPP can serve as a reliable confidence score for synthesized speech segments, which can then be aggregated into a meaningful overall intelligibility metric.

Key Results

  • High Correlation with Subjective Scores: The proposed TCGPP-based objective intelligibility measure achieved a correlation coefficient of 0.9 with human subjective intelligibility scores on the Blizzard Challenge 2010 EH1 task.
  • Superior to ASR Baseline: This correlation is significantly higher than the 0.71 achieved by the accuracy of a standard ASR system (word recognition rate) used as a baseline.
  • Accurate Ranking: The proposed measure successfully replicated the rank ordering of TTS systems as determined by human perception, both in the internal voice font experiment and the Blizzard Challenge.
  • Internal Voice Font Comparison: In an experiment comparing three internal TTS voices, the TCGPP measure produced an ordering consistent with human perception (SUR) and significantly better than ASR's WRR.

Limitations and Future Work

  • Computational Cost: While objective, the TCGPP calculation involves complex matching and probability computations, which could be computationally intensive. The paper mentions "time relaxation registration" and "weighted acoustic and language model likelihood" as factors handled, implying some optimization, but detailed computational complexity is not a primary focus.
  • Dependence on ASR and Acoustic Models: The accuracy of the proposed method relies on the quality of the underlying ASR system and its acoustic model. Errors in these components could propagate to the intelligibility assessment.
  • Future Work: The authors suggest that the TCGPP scores can be summarized into a single metric to compare and rank multiple TTS systems, which they have demonstrated. Further exploration into tailoring templates for specific types of TTS errors or languages could be a future direction. The paper also implies that the raw objective scores can be linearly transformed to any specific range, suggesting flexibility in reporting.

Summary updated

Abstract

Speech intelligibility is one of the most important measures in evaluating text-to-speech (TTS) synthesizer. For fast comparing, developing, and deploying TTS systems, automatic objective intelligibility measurement is desired, as human listening test is label intensive, inconsistent, and with expensive cost. In this work, we propose an automatic objective intelligibility measure for synthesized speech using template constrained generalized posterior probability (TCGPP). TCGPP is a posterior probability based confidence measure, which has the advantage to identify errors in synthesized speech at small granularity level. Moreover, the TCGPP scores over a testing set can be summarized into an overall objective intelligibility metric to compare two synthesizers, or rank multiple TTS systems. We conducted the experiments using the synthesized test sentences from all the participants of EH1 English task in Blizzard Challenge 2010. The results show the proposed measure has high correlation (corr=0.9) with subjective scores and ranking.
BibTeX
@inproceedings{wang2012objective,
  author = {Linfang Wang and Lijuan Wang and Yan Teng and Zhe Geng and Frank K. Soong},
  title = {{Objective Intelligibility Assessment of Text-to-Speech System using Template Constrained Generalized Posterior Probability}},
  booktitle = {Interspeech},
  year = {2012},
  url = {https://openreview.net/forum?id=JSsrs5P8Zw},
}
Figure