Subjective experiments on influence of response timing in spoken dialogues

Toshihiko Itoh, Norihide Kitaoka, Ryota NishimuraInterspeech 2009

TL;DR

This paper investigates how the timing of responses in spoken dialogues affects naturalness and comprehension. Through subjective experiments with synthesized and real speech, they found that while precise timing based on utterance content is important, it's not the sole determinant of naturalness. The interplay between response timing and other acoustic features like emotion and intonation is crucial for creating truly human-like interactions.

What Problem Does This Paper Address?

Creating natural and smooth human-machine communication through spoken dialogue systems is a significant challenge. While speech recognition and synthesis have advanced, the interaction often feels less natural than human-to-human conversations. Previous research by the authors suggested that "dialogue rhythm," encompassing response timing, pitch (F0), and speech rate, plays a vital role in achieving this naturalness. However, their earlier systems, while improving rhythm, still lacked human-like smoothness. This paper aims to verify their previous findings on response timing and explore how it interacts with other acoustic cues, particularly in the context of task-oriented dialogues, which were less explored than free conversations. They hypothesize that listeners have an internal model of general response timings and that timings conforming to utterance content are perceived as most natural.

Methods

The researchers conducted two subjective experiments.

Experiment 1 (Synthesized Speech): * Dialogue Generation: Very short, four-turn task-oriented dialogues were synthesized. The core manipulation was varying the response timing – the pause duration between the end of one utterance and the start of the next. * Response Timing Patterns: Seven distinct patterns were tested, including: * Average timings based on dialogue acts. * Average timings based on specific utterance contents. * Random applications of content-based timings. * Fixed timings (1 second, 0.5 seconds, and -0.3 seconds, indicating overlap). * Speech Synthesis: The FineSpeech SDK V2.1 was used. To ensure clear distinctions, speaker A was synthesized with a female voice and speaker B with a male voice, keeping other parameters (pitch, emotion, intonation, speech rate) at default. * Subjective Evaluation: Test subjects rated dialogues on a 7-point scale for naturalness and synthesized speech quality. They also provided qualitative impressions of the speakers. Two groups of subjects participated, with one group specifically instructed to focus on utterance content and response timings.

Experiment 2 (Real and Synthesized Speech): * Dialogue Selection: A lively, informal conversation extract from the CSJ corpus (Corpus of Spontaneous Japanese) was used. * Speech Sample Creation: Four types of speech samples were created from this extract: 1. Real Speech (Original Timing): Actual speech with unaltered response timings. 2. Real Speech (Fixed Timing): Actual speech with all response timings fixed at 0.5 seconds. 3. Synthesized Speech (Original Timing): Synthesized speech using the original response timings. 4. Synthesized Speech (Fixed Timing): Synthesized speech with all response timings fixed at 0.5 seconds. * Speech Synthesis: The Galatea Talk software was used for synthesis. Default parameters for pitch, emotion, intonation, and speech rate were employed. The original dialogue was between two females, but male and female speakers were used for synthesis to better distinguish speakers. Overlapping parts were preserved with their original timing. * Subjective Evaluation: Similar to Experiment 1, subjects rated naturalness, agreement between prosodic features and content, ease of understanding intention/emotion, synthesized speech quality, and ease of comprehension.

Datasets

  • CSJ corpus (Corpus of Spontaneous Japanese): Used in Experiment 2 as the source for real speech samples. This corpus is provided by the National Institute for Japanese Language.

Key Insights and Ideas

  • Response Timing Matters, But Isn't Everything: The experiments strongly suggest that a natural response timing does exist and significantly impacts dialogue naturalness. However, simply fixing timing to an average or even content-based value isn't a silver bullet.
  • The "Sweet Spot" of Timing: In Experiment 1, a fixed 0.5-second response time (sample 1-6) received the highest naturalness scores, even outperforming timings derived from detailed utterance content analysis (sample 1-2). This hints at a balance point.
  • Overlaps Can Be Tricky: While overlaps are natural in human speech, synthesized speech with overlaps (especially in Experiment 2) was rated poorly, indicating that simply replicating timing isn't enough for natural overlap.
  • The Crucial Role of Non-Linguistic Cues: Experiment 2 revealed that when non-linguistic information (emotion, intonation, speech rate) conforms to utterance content in real speech, content-based response timings are preferred. Conversely, with synthesized speech, a fixed timing (0.5s) was better because it provided a consistent, albeit machine-like, balance of these non-linguistic cues. This suggests that naturalness arises from the harmony of timing and acoustic features, not just timing alone.
  • Machine-like vs. Human-like Cues: The authors posit that the machine-like default settings for emotion, intonation, and speech rate in Experiment 1, combined with natural response timings, might have led to the perception of "calm" and "relaxed" speakers, whereas content-based timings might have sounded more "human" but also "tense" or "impatient" due to the mismatch with synthesized prosody.

Key Results

  • Experiment 1:
    • Speech sample 1-6 (fixed 0.5s timing) received the highest naturalness scores.
  • Speech sample 1-2 (content-based timing) received the second-highest scores.
  • A strong correlation was found between the naturalness of response timing and the perceived quality of synthesized speech.
  • Overlapping speech samples (especially 1-7) received the lowest scores.
  • Response timings aligned with dialogue acts were evaluated as more natural than random timings.
  • Experiment 2:
    • Real Speech: When non-linguistic information was congruent with content, content-based response timings (2-1) were rated higher than fixed 0.5s timings (2-2) across naturalness, speech agreement, and intention.
  • Synthesized Speech: Fixed 0.5s timings (3-2) generally outperformed original timings (3-1) in naturalness, quality, and comprehension, suggesting that a consistent, albeit artificial, balance of acoustic features is better than mismatched original timings with synthesized prosody.

Limitations and Future Work

  • Limitations:
    • The synthesized speech in Experiment 1 used default settings for emotion, intonation, and speech rate, which may have skewed results.
  • The evaluation of overlaps in synthesized speech was poor, indicating a need for more sophisticated handling.
  • It's challenging to fully estimate emotion and dialogue acts from response timing alone.
  • Future Work:
    • Conduct further subjective experiments using tools to manipulate specific acoustic characteristics (e.g., voice changers) to isolate their impact.
  • Analyze acoustic characteristics necessary for natural-sounding overlaps in synthesized speech.
  • Develop models that consider speech synthesis performance alongside response timing.
  • Investigate how response timing can provide appreciable information about emotion and dialogue acts.

Summary updated

BibTeX
@inproceedings{itoh2009subjective,
  author = {Toshihiko Itoh and Norihide Kitaoka and Ryota Nishimura},
  title = {{Subjective experiments on influence of response timing in spoken dialogues}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure