A framework of reply speech generation for concept-to-speech conversion in spoken dialogue systems

Seiya Takada, Yuji Yagi, Keikichi Hirose, Nobuaki MinematsuInterspeech 2007

TL;DR

This paper introduces a novel framework for concept-to-speech (CTS) conversion in spoken dialogue systems, specifically for generating natural-sounding reply speech. By preserving linguistic information like syntactic structure and word importance throughout the generation process using "tag LISP forms," their system achieves more prosodically accurate and natural speech compared to standard TTS approaches.

What Problem Does This Paper Address?

Current spoken dialogue systems often rely on off-the-shelf text-to-speech (TTS) synthesizers. While effective for generating speech from text, these systems struggle to incorporate the rich linguistic information (syntactic structure, discourse context, word importance) that is readily available during the dialogue's reply sentence generation phase. This leads to reply speech that can sound unnatural or fail to convey the intended emphasis, hindering effective human-machine interaction. The authors aim to bridge this gap by developing a unified framework that directly converts a "concept" (representing the intended meaning and linguistic nuances) into speech, ensuring that higher-level linguistic information is reflected in the prosody of the output.

Methods

The core of their approach is a Concept-to-Speech (CTS) framework that operates in a unified process. Key techniques include:

  • Tag LISP Forms: Concepts are represented using LISP forms where "tags" replace words or phrases. These tags carry attributes like "importance" and "novelty," which are crucial for prosodic control. This structure inherently preserves syntactic relationships.
  • Phrase and Sentence Templates: Sentence generation involves filling in these tag LISP forms using phrase and sentence templates. This allows for flexible generation of various sentence styles, from simple to complex/compound, by manipulating how phrases are combined.
  • Prosodic Control using Fo Model: The system utilizes a phrase/accent command model (Fo model) to control the fundamental frequency (Fo) contour of the generated speech. These commands, derived from the tag LISP forms, directly correspond to linguistic information.
  • Phrase/Accent Command Symbols: Specific symbols are inserted into the phonetic representation of the sentence to indicate phrase and accent command onsets and magnitudes. These symbols are assigned based on word importance, novelty, and syntactic position, drawing from regression analysis of dialogue speech prosody.
  • JUMAN+KNP Analysis (for comparison): For evaluation, they compare their method against a system that uses syntactic structures derived from external Japanese parsers (JUMAN and KNP) but without explicit discourse information.

Datasets

The paper does not explicitly mention training datasets. For evaluation, they used eight sentences from their road guidance dialogue system, synthesized using three different methods. These synthesized speeches were then evaluated by 24 Japanese speakers.

Key Insights and Ideas

  • Linguistic Information is Key for Natural Prosody: The most significant insight is that directly incorporating linguistic information (syntactic structure, word importance, novelty) into the speech generation process, rather than relying solely on text, dramatically improves the naturalness of reply speech.
  • Tag LISP Forms as a Unified Representation: The use of tag LISP forms proves to be an effective way to maintain syntactic structure and encode semantic/discourse information throughout the entire generation pipeline, from concept to speech.
  • Flexibility through Templates: The combination of phrase and sentence templates, powered by tag LISP forms, allows for generating a wide variety of sentence styles without a proportional increase in the number of templates.
  • Importance and Novelty Matter: The "importance" and "novelty" attributes assigned to words within the tag LISP forms are directly linked to prosodic variations, making the speech more informative and engaging.

Key Results

A listening experiment involving 24 Japanese speakers evaluated the naturalness of synthesized speech. The results showed:

  • The proposed method consistently received the highest average scores across all tested sentences, indicating superior prosodic quality.
  • The proposed method was significantly better than the "No discourse information" baseline.
  • While the proposed method also outperformed the JUMAN+KNP analysis method, the difference was less pronounced in some cases, suggesting that while syntactic information is important, discourse information (importance/novelty) provides an additional significant boost.
  • The advantage of the proposed method tended to increase with sentence length, implying its robustness for more complex utterances.

Limitations and Future Work

  • Limited Discourse Information: The authors acknowledge that the current assignment of "importance" and "novelty" is basic. Future work aims to improve the assignment of these degrees of importance and novelty.
  • Prosodic Control Refinement: Further refinement of prosodic control is planned through more in-depth analysis of dialogue in the road-guidance context.
  • Style Adaptation: The authors envision developing a scheme to change the style of reply sentences based on user preference and the ongoing dialogue situation.
  • Future Work on Dialogue Analysis: Better prosodic control is planned through analysis of the dialogue in the road-guidance situation.

Summary updated

BibTeX
@inproceedings{takada2007framework,
  author = {Seiya Takada and Yuji Yagi and Keikichi Hirose and Nobuaki Minematsu},
  title = {{A framework of reply speech generation for concept-to-speech conversion in spoken dialogue systems}},
  booktitle = {Interspeech},
  year = {2007},
}
Figure