Talking heads for interacting with spoken dialog smart-home systems

Christine Kühnel, Benjamin Weiss, Sebastian MöllerInterspeech 2009

TL;DR

This paper investigates whether using a "talking head" avatar instead of just voice output improves user experience and perceived quality in smart-home systems. They found that while a talking head can be preferred, its quality is highly dependent on visual appeal, speech clarity, and crucially, synchronization between voice and lip movements. Ultimately, the benefits of a talking head in this specific smart-home context were not as pronounced as expected, suggesting that the application domain and potential distractions play a significant role.

What Problem Does This Paper Address?

As smart-home systems become more sophisticated, developers are exploring multimodal interfaces to enhance user interaction. A key question is whether adding a visual component, specifically an embodied conversational agent (ECA) like a "talking head," improves the overall user experience and perceived system quality compared to traditional voice-only interfaces. While ECAs have shown promise in other domains, their effectiveness in the specific context of smart-home control, where interactions might be command-driven and less conversational, is less clear. This paper aims to bridge this gap by empirically evaluating the impact of talking heads on smart-home system quality, focusing on user perception and efficiency.

Methods

The researchers designed an experiment using a functional smart-home living room equipped with controllable devices (TV, lights, blinds, phone). The core of the experiment involved the INSPIRE spoken dialog system, which was configured with six different output "metaphors": two types of talking heads (Thinking Head and MASSY) and voice-only output. The talking heads were created by combining two animated head models with two different speech synthesis systems (Mary and Mbrola).

Participants interacted with the system to complete various tasks within three scenarios. The experiment employed a mixed-design approach: the voice synthesis system was varied between participants, while the output metaphor (talking head vs. voice-only) was varied within participants. After each scenario, participants completed questionnaires assessing various quality dimensions. In addition to subjective ratings, objective measures like dialog duration were logged to assess efficiency.

Datasets

Not discussed in the paper. The experiment used a custom-built smart-home environment and a spoken dialog system (INSPIRE).

Key Insights and Ideas

  • The "Persona Effect" Isn't Guaranteed: While the "persona effect" (positive impact of life-like agents) is a known phenomenon, this study suggests it doesn't automatically translate to smart-home systems. The specific nature of the interaction (command-driven, potentially less conversational) and the presence of visual distractions (like a TV screen) might diminish the perceived benefits of a talking head.
  • Synchronization is King (for Talking Heads): The study strongly reinforces the idea that for talking heads to be effective, the synchronization between the spoken audio and the lip movements of the avatar is paramount. Poor synchronization can negate any potential advantages.
  • Visual and Speech Quality Matter, but Fit is Less Critical: Both the visual quality of the head and the quality of the synthesized speech significantly impact the overall perceived quality of a talking head. However, the "fit" between the voice and the head (e.g., ensuring a male voice with a male-looking head) was found to be less critical, likely because common-sense expectations were met.
  • Efficiency Trade-offs: Talking heads, particularly animated ones, can increase dialog duration due to processing time for animation. While this didn't negatively impact overall quality ratings in this study, it's a practical consideration for system design.

Key Results

  • Hypothesis 1 (Preference for Talking Head): Not confirmed. Participants did not show a clear preference for talking heads over voice-only output in this smart-home context.
  • Hypothesis 2 (Talking Head Quality Impacts System Quality): Confirmed. Higher perceived quality of the output component (talking head or voice-only) led to higher perceived overall system quality. Specifically, the MASSY talking head, which had lower overall quality ratings, resulted in lower system quality ratings compared to voice-only and the Thinking Head.
  • Hypothesis 3 (Factors Influencing Talking Head Quality): Confirmed. Visual quality, speech quality, and synchronization of voice and lip movements were found to be significant factors influencing the overall quality of talking heads.
  • Quantitative Findings:
    • The Thinking Head (TH) received higher OVERALL QUALITY (M=0.76) and VISUAL QUALITY (M=0.49) ratings than MASSY (M=0.31 and M=-0.06, respectively).
  • The Mary voice was preferred over the Mbrola voice for SPEECH QUALITY (M=0.82 vs. M=0.46).
  • Dialog duration was longer for talking heads (TH: M=188.24s, MASSY: M=290.82s) compared to voice-only (M=160.49s).
  • SYSTEM QUALITY was rated higher when using voice-only and TH compared to MASSY.

Limitations and Future Work

  • Limited Dialog Complexity: The authors acknowledge that the smart-home interaction in their experiment was relatively simple and command-driven. They suggest that talking heads might offer greater benefits in more complex, conversational applications within the smart-home domain, such as information or companionship services.
  • Visual Distractions: The presence of a TV screen in the experimental setup may have diverted user attention from the talking head, potentially masking its benefits.
  • Privacy Concerns: The paper briefly touches upon the possibility that privacy concerns (users feeling observed by an agent) could outweigh the advantages of a talking head, a factor not directly investigated.
  • Future Work: The authors plan to further analyze their audio recordings to gain deeper insights into the interrelations between talking head quality and system quality. They also suggest exploring scenarios where the "persona effect" might be more pronounced.

Summary updated

BibTeX
@inproceedings{kuhnel2009talking,
  author = {Christine Kühnel and Benjamin Weiss and Sebastian Möller},
  title = {{Talking heads for interacting with spoken dialog smart-home systems}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure