Integrated Natural Spoken Dialogue System of Jijo-2 Mobile Robot for Office Services

AAAI 1999

TL;DR

This paper introduces Jijo-2, a mobile robot designed for office services, showcasing an integrated natural spoken dialogue system. It tackles the challenges of noisy environments and fragmented human speech in Japanese, enabling Jijo-2 to understand and respond to user requests for navigation, information retrieval, and task execution.

What Problem Does This Paper Address?

As robots move beyond structured factory floors into dynamic office environments, they need more sophisticated human-robot interaction capabilities. Traditional robots often rely on explicit commands, but office dwellers are typically non-experts who expect more natural, flexible communication. This paper addresses two key challenges in creating such natural spoken dialogue systems for robots:

  1. Noisy Environments: Offices are inherently noisy, which significantly degrades the performance of speech recognition systems.
  2. Fragmented and Omitted Speech: Japanese, in particular, frequently omits subjects, objects, and pronouns from sentences when context makes them clear. This makes it difficult for a system to fully understand the user's intent from short, incomplete utterances.

The goal is to enable robots like Jijo-2 to understand and act upon user requests for services such as answering queries about people's locations, providing route guidance, and executing tasks like sending emails or faxes.

Methods

The Jijo-2 system employs several innovative techniques to overcome the aforementioned challenges:

  • Noise-Free Voice Acquisition: A microphone array with eight omni-directional microphones is used. By employing a delay-and-sum beamforming method, the system can pinpoint the direction of the speaker and focus on their voice, effectively suppressing ambient noise. This is crucial for maintaining good speech recognition performance.
  • Robust Speech Recognition:
    • Multiple Voice Recognizers: The system utilizes multiple speech recognition processes, each with different dictionaries and grammars. This allows the dialogue manager to select the most appropriate recognizer based on the current context, improving accuracy.
  • HMM-Based Recognizer (NINJA): A speaker-independent, Hidden Markov Model (HMM)-based Japanese speech recognizer is used. It searches for word symbols that match a predefined grammar, outputting recognized words along with confidence scores and sound source direction.
  • Handling Fragmented Utterances and Zero Pronouns:
    • Dialogue Contexts: The system defines five dialogue contexts (query, update, identification, navigation, call) to manage the flow of conversation and track required information (slots).
  • Attentional State Manager: This is a core component for handling under-specified input. It maintains an ordering of discourse referents based on their salience. This helps in resolving ambiguities and filling in missing information (like zero pronouns) by inferring the most likely referent based on conversational history and linguistic cues. The paper specifically mentions adapting the centering theory for Japanese, which ranks entities based on syntactic function and empathy.
  • Semantic Frame Construction: The system constructs semantic frames from fragmented utterances by combining information from the dialogue context and the attentional state.
  • Integrated Architecture: Jijo-2 features an event-driven, multi-agent architecture where various modules (speech recognition, navigation, vision, database interaction) communicate over TCP/IP. This allows for concurrent behaviors and flexible integration of software modules.

Datasets

Not explicitly discussed in the paper. The speech recognition system (NINJA) is described as using a "phonetic dictionary created by the HMM learning beforehand," implying it was trained on relevant Japanese speech data, but specific datasets are not detailed.

Key Insights and Ideas

  • Beamforming for Real-World Robotics: The paper demonstrates the practical effectiveness of microphone arrays and beamforming for achieving robust speech recognition in noisy, real-world environments like offices, a significant improvement over single-microphone systems.
  • Attentional State as a Key to Natural Japanese Dialogue: The concept of an attentional state manager is highlighted as crucial for handling the inherent ambiguity and pronoun omission in Japanese. By tracking conversational focus, the robot can infer missing information and generate more natural responses.
  • Dialogue Contexts for Task Management: Defining specific dialogue contexts (e.g., navigation, query) provides a structured way to guide conversations and ensure all necessary information (slots) is gathered before executing a task.
  • Integration is Key: The success of Jijo-2 lies in the seamless integration of its dialogue system with its navigation, vision (face recognition), and database capabilities, allowing it to perform complex office services.
  • Safety First: The dialogue state transitions include mechanisms for handling commands like "stop" and "cancel" with high priority, ensuring user safety.

Key Results

  • Speech Recognition Improvement: The microphone array system showed a significant reduction in noise, improving recognition performance in various noisy conditions compared to a single microphone.
  • Demonstration Success Rate: Approximately 90% of demonstrations of Jijo-2 performing office tasks were successful.
  • Utterance Misunderstanding Rate: During demonstrations, an average of 10-20% of utterances were misunderstood. However, the system's structured dialogue states and confirmation mechanisms prevented catastrophic failures.
  • Naturalness: The use of attentional states and dialogue contexts enabled the robot to handle under-specified utterances and zero pronouns, leading to smoother and more natural Japanese dialogue.

Limitations and Future Work

  • Quantitative Evaluation: The authors acknowledge that a thorough quantitative evaluation of the entire system's performance and user-friendliness is difficult due to the challenges of controlling environments for fair comparisons.
  • Speech Recognition Errors: While improved, speech recognition still experienced an average of 10-20% misunderstanding rate in demonstrations, primarily due to unclear pronunciation or sudden loud noises.
  • Multi-User Dialogue: A significant future goal is to enable dialogue with more than one user simultaneously. This will require advanced capabilities to distinguish speakers using visual cues, voice direction, and even voice characteristics.
  • More Sophisticated Integration: Further integration of dialogue with other sensory inputs and more complex reasoning capabilities is envisioned for building truly intelligent artificial creatures.

Summary updated

BibTeX
@inproceedings{1999integrated,
  title = {{Integrated Natural Spoken Dialogue System of Jijo-2 Mobile Robot for Office Services}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1999},
}
Figure