Abstract
This paper presents an experimental multimodal interface that integrates a 3D animated face with a continuous speech dialogue system. By employing facial expressions as functional communicative signals—categorized into syntactic, speaker, and listener displays—the system aims to reduce the cognitive load and "mental barrier" for users during human-computer interaction. Experimental results with 32 subjects indicate that these nonverbal cues are particularly effective during initial interactions, helping users build better mental models of the system's internal state.
Introduction
In the early 1990s, human-computer dialogue was often a high-stress experience due to the lack of nonverbal feedback. While human communication relies on multiple channels including voice, gaze, and facial expressions, computers remained largely unresponsive to these cues. A critical gap identified by researchers was the absence of "backchannel" feedback—the subtle nods or eyebrow movements that signal understanding or confusion. Without these, users often feel uncertain about whether a system is processing their input or has stalled. This research explores using an anthropomorphic interface to lower the complexity of interacting with machines.
System Architecture
The experimental system consists of two primary subsystems connected via Ethernet: a Facial Animation Subsystem and a Speech Dialogue Subsystem.
The Facial Animation Subsystem generates a 3D head composed of approximately 500 polygons. It utilizes a "muscle model" to simulate realistic movements such as jaw rotations and eyebrow flashes. The Speech Dialogue Subsystem manages the core logic, utilizing Hidden Markov Models (HMM) for continuous speech recognition, a 24-rule grammar for syntactic analysis, and a plan recognition module to infer user intent (e.g., inquiries about Sony computer products).
Mapping Conversational States to Facial Displays
A key innovation of this work is the systematic mapping of conversational situations to specific facial "displays." These displays are not merely emotional but serve functional roles in the dialogue.
The researchers categorized expressions into three functional roles:
- Syntactic Displays: Marking the structure of a sentence (e.g., eyebrow raising for a question).
- Speaker Displays: Illustrating the idea being conveyed.
- Listener Displays: Providing real-time feedback while the user is speaking (e.g., "Indication of Attendance").
For instance, if the speech recognizer identifies multiple candidates with close scores, the system displays a "Moderately Confident" expression (eyebrow raising). If the user's utterance is syntactically invalid, the face shows a "Not Confident" expression (eyebrow lowering), providing immediate, non-verbal feedback.
Experimental Evaluation and Results
The system was evaluated using 32 volunteer subjects divided into two groups to test the "first contact" effect. One group interacted with the facial system first, while the other began with a text-only interface.
The results highlighted several key insights:
- Social Lubrication: Facial displays act as a "social lubricant," significantly reducing the mental barrier and stress associated with computing.
- Mental Model Formation: Users who started with the facial interface performed better even when later switched to a text-only version. The visual feedback helped them more quickly grasp the system's logic and limitations.
- Functional Feedback: "Smooth" conversations were characterized by frequent "Indication of Attendance" displays, confirming that the system was actively listening.
Conclusion
This research demonstrates that integrating facial animation into speech dialogue systems provides a more natural and less stressful user experience. By treating facial expressions as "situated actions," the system provides immediate feedback that voice alone cannot achieve. While limited by its narrow domain and specific language (Japanese), this work laid the groundwork for modern multimodal AI avatars, envisioning a future where human-computer interaction is a truly synergic, multi-channel exchange.
Summary updated