Abstract
This paper presents a prototype for an "Active Agent Oriented Multimodal Interface," a system designed to move beyond passive, reactive human-computer interaction. By integrating real-time face recognition, speech processing, and a 3D animated avatar, the researchers developed an agent that proactively initiates dialogue, maintains eye contact, and personalizes interactions based on the user's identity. Evaluation results demonstrate high accuracy in both visual and auditory recognition, suggesting a viable path toward more natural digital assistants.
Introduction
In the mid-1990s, human-computer interaction (HCI) was largely unimodal and reactive, failing to capture the rich, multimodal nature of human communication. Human interaction relies on the simultaneous use of voice, facial expressions, and eye contact to manage conversational flow and convey meaning. This research addresses the psychological need for "intimacy" and engagement in HCI by creating an agent that behaves like a human partner. The primary challenge lay in integrating disparate technologies—computer vision, speech recognition, and 3D graphics—into a cohesive, real-time system capable of operating in unconstrained office environments.
System Architecture
The system's core is an Interaction Manager that coordinates five specialized subsystems to ensure synchronized responses to visual and auditory cues.
- Facial Display Sub-system: Utilizes a 3D model of approximately 500 polygons. Texture mapping from real photographs provides a realistic appearance, while expressions are controlled via Action Units based on the Facial Action Coding System (FACS).
- Vision Sub-system: Acts as the agent's "eyes," identifying users through higher-order local autocorrelation features and Linear Discriminant Analysis (LDA). This method is robust enough to function without specialized backgrounds.
- Speech Recognition Sub-system: A speaker-independent, continuous speech recognizer designed to process short, spontaneous sentences.
- Speech Synthesis Sub-system: Provides the agent with a male voice for verbal responses.
- Interaction Manager: Resolves timing and logic, ensuring the agent's behavior is contextually appropriate.
Experimental Methodology and Datasets
The system was evaluated using two primary internal datasets:
- Face Recognition: A dataset of 116 individuals was used to test identification accuracy. The learning phase required roughly 50 images per user alongside background samples.
- Speech Recognition: Data from 40 subjects provided 183 spontaneous utterances to evaluate the robustness of the dialogue system in real-world scenarios.
Key Insights and Results
The researchers identified several critical factors for successful multimodal interaction. Proactivity was found to be essential; by initiating conversation when a user is detected, the agent breaks the traditional passive software mold. Furthermore, eye contact—implemented via 3D eyeball tracking—was vital for making the agent feel "friendly" rather than robotic.
Performance metrics for the prototype were strong for the era:
- Vision Accuracy: The system achieved a 98% recognition rate for 116 individuals, operating at 5 frames per second on a SUN Sparcstation 10.
- Speech Accuracy: Recognition of spontaneous speech reached 84.2%.
- Graphics Performance: The 3D face rendered at 13 frames per second on an SGI Indigo2, providing smooth, human-like animation.
Because the vision system identifies the specific user, the agent can perform identity-aware tasks, such as delivering personalized E-mail notifications or managing individual schedules.
Conclusion and Future Work
While the prototype successfully demonstrated the potential of active multimodal agents, several limitations remain. The current system lacks sophisticated error recovery, meaning it cannot easily detect when its recognition modules have failed. Additionally, the vocabulary is limited to approximately 100 words, and the vision system is optimized for single-user frames. Future iterations will focus on expanding the agent's intelligence, increasing its vocabulary, and incorporating hand gestures to further enhance non-verbal communication.
Summary updated