Expressing speaker's intentions through sentence-final intonations for Japanese conversational speech synthesis

Kazuhiko Iwata, Tetsunori KobayashiInterspeech 2012

TL;DR

This paper explores how subtle variations in the pitch contours at the end of Japanese sentences can convey different speaker intentions, moving beyond simple rising or falling tones. By analyzing and synthesizing speech with these nuanced intonations, the researchers demonstrate their crucial role in natural conversational speech synthesis, paving the way for more expressive and human-like AI interactions.

What Problem Does This Paper Address?

Current text-to-speech (TTS) systems, while functional for reading out information, often lack the expressiveness needed for natural conversational interactions. Spoken dialogue systems, like those in robots or voice assistants, require the ability to convey not just information but also the speaker's intentions, attitudes, and subtle emotional nuances. In Japanese, sentence-final particles and auxiliary verbs play a significant role in signaling these intentions, and these linguistic cues are closely tied to the intonation patterns at the end of a sentence. While the importance of sentence-final intonation in Japanese has been recognized, there's a gap in understanding and implementing the subtle variations that convey specific speaker intentions, which this paper aims to bridge.

Methods

The researchers employed a multi-step approach:

  1. Data Collection and Preprocessing: They utilized a corpus of approximately 2,000 Japanese utterances from a conversational robot context. To focus on sentence-final intonation, they extracted the fundamental frequency (F0) contours from the final vowel of sentences where the vowel was not devoiced.
  2. F0 Contour Normalization: To make contours comparable across different speakers and utterances, they normalized both the time and frequency axes. This involved approximating the logarithmic F0 contour with a third-order least squares curve, sampling it at 11 points, and then parallel translating the starting point to the origin.
  3. Clustering: The normalized F0 contours were then clustered using Ward's algorithm to identify distinct intonation patterns. This revealed a variety of contours beyond simple rising and falling, including rise-fall and fall-rise patterns.
  4. Subjective Evaluation: To understand the perceptual impact of these intonations, they selected 10 representative F0 contours from the clustering results. These contours were then used to synthesize six Japanese sentences, each designed to have multiple meanings depending on the intonation. Nine listeners were asked to identify the speaker's intention from a list of options or by free description.

Datasets

The study used a custom dataset of approximately 2,000 Japanese sentence utterances, originally collected for a conversational robot's dialogue system. This dataset was specifically chosen for its relevance to natural conversational speech. No standard public benchmarks were explicitly mentioned as being used for training or evaluation, but the paper implies the creation of a benchmark for evaluating sentence-final intonation perception.

Key Insights and Ideas

  • Beyond Simple Tones: The research highlights that sentence-final intonations in Japanese are far more nuanced than just rising or falling. Complex contours like rise-fall and fall-rise play distinct roles in conveying meaning.
  • Subtlety Matters: Even minor differences in the shape of an F0 contour can lead to significantly different perceived intentions. For example, a slight variation in the descent of a falling tone could shift perception from "conviction" to "doubt."
  • Contextual Meaning: The same sentence can convey different intentions based on the intonation. This underscores the importance of prosody in disambiguating meaning in spoken language.
  • Intention Mapping: The study provides a preliminary mapping between specific F0 contour shapes and perceived intentions like "conviction," "doubt," "inquiry," "surprise," "imperative," "desire," and "hearsay."

Key Results

The subjective evaluation revealed significant correlations between specific sentence-final F0 contours and perceived speaker intentions. For instance:

  • Sentences (S1), (S2), (S3): Rising intonations were generally perceived as interrogative, while falling intonations indicated conviction. Contour C1 conveyed "conviction," while C2, with a more gradual descent, suggested "doubt."
  • Sentence (S4): Flat intonations (C6, C2) indicated "disinterest," while a deep fall-rise (C10) conveyed "imperative."
  • Sentences (S5) and (S6): Rising intonations primarily signaled "inquiry," whereas falling or flat intonations suggested "hearsay" or "acceptance."

The paper notes that while some contours conveyed similar intentions, they could still evoke different impressions, suggesting that the quality of the intonation (e.g., steepness, smoothness) is crucial.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Focus on F0: This study primarily focused on the fundamental frequency (F0) contours. However, other prosodic features like duration and power are also crucial for expressing speaker intentions and attitudes. Future work should investigate the interplay of these features.
  • Limited Corpus: The dataset, while relevant, was derived from a specific context (conversational robot). Expanding the corpus to include a wider range of conversational scenarios and speakers would enhance generalizability.
  • Perceptual Nuances: While the study identified distinct intentions, it also noted that some contours that conveyed similar intentions still gave different impressions. Further research is needed to understand these subtle perceptual differences.
  • Japanese-Specific: The findings are specific to Japanese intonation patterns. Cross-linguistic studies would be valuable to understand universal vs. language-specific prosodic cues for intention.
  • Integration into TTS: The ultimate goal is to integrate these findings into conversational speech synthesis systems to create more natural and expressive AI interactions.

Summary updated

BibTeX
@inproceedings{iwata2012expressing,
  author = {Kazuhiko Iwata and Tetsunori Kobayashi},
  title = {{Expressing speaker's intentions through sentence-final intonations for Japanese conversational speech synthesis}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure