A Spoken Dialogue Interface to a Geologist’s Field Assistant

John Dowding, James HieronymusNAACL 2003

TL;DR

This paper presents a spoken dialogue interface for a robot assistant designed to help geologists explore planetary surfaces. The system tackles the challenges of noisy environments, open microphones, and voice annotation recording, demonstrating improved accuracy and speed with a probabilistic context-free grammar.

What Problem Does This Paper Address?

Exploring planetary surfaces, like Mars, presents a significant challenge for geologists. They need to collect samples, document their findings, and track their progress, all while wearing bulky spacesuits that can interfere with communication and data recording. Traditional methods rely on manual logging, which is cumbersome and prone to error in such demanding environments. This paper addresses the need for a hands-free, intelligent assistant that can understand and respond to a geologist's spoken commands and annotations, even in the presence of background noise and the inherent limitations of open-microphone systems. The core technical challenges lie in accurately recognizing speech intended for the system amidst other sounds and in effectively managing voice annotations as data objects.

Methods

The core of the system is a spoken dialogue interface that leverages a grammar-based language model for speech recognition. A key innovation is the automatic derivation of this grammar from a Unification Grammar, which is also used for parsing and interpretation. This ensures consistency between what the system understands and how it processes the information.

For handling voice annotations, the system temporarily increases the speech end-pointing interval. This allows for recording multi-sentence voice notes without treating pauses between sentences as the end of the annotation. Recording is triggered by specific speech acts, such as "Take a voice note" or "Annotate sample bag one."

The paper highlights the transition from a baseline Context-Free Grammar (CFG) language model to a Probabilistic Context-Free Grammar (PCFG). This probabilistic approach, trained on collected data, aims to improve the system's ability to discriminate between in-domain utterances (intended for the system) and out-of-domain utterances (intended for other listeners or irrelevant noise).

Datasets

The primary dataset for this work was collected during two field tests in 2002:

  • A one-week rehearsal at NASA Ames Research Center (JSC) in May.
  • A two-week field test in the Arizona desert in September, near the Petrified Forest National Park and Meteor Crater.

During the September tests, approximately 5,000 sound files were recorded from 8 subjects, including those in spacesuits and those in shirt-sleeve conditions with high wind. Of these, 1059 wave files were transcribed. After filtering out files with no speech or out-of-domain speech, 608 within-domain utterances were obtained, which were split into 80% for training and 20% for testing. The Probabilistic CFG was built using only 485 utterances from the training data.

Key Insights and Ideas

  • Speech as Data Annotation: The system treats voice notes not just as audio recordings but as structured data objects that can be associated with other information (e.g., sample bags). This is crucial for scientific fieldwork where detailed, contextualized notes are vital.
  • Robustness to Noise and Open Mics: The primary technical hurdle is achieving reliable speech recognition in a noisy, open-microphone environment typical of field operations. The system's ability to discriminate between intended commands and ambient sounds is a key focus.
  • Probabilistic Grammars for Discrimination: Moving from a deterministic CFG to a PCFG significantly improves the system's ability to distinguish between relevant and irrelevant speech, reducing false acceptances of out-of-domain utterances.
  • Efficiency Gains: The probabilistic approach not only improves accuracy but also leads to substantial improvements in recognition speed, making the system more practical for real-time interaction.

Key Results

The paper presents performance metrics comparing a baseline CFG language model with a Probabilistic CFG language model trained on collected data.

Baseline CFG Language Model (September Test): * Word Error Rate (WER): 12.56% * False Rejection Rate (FREJ): 4.54% * Adjusted Word Error Rate (AWER): 7.72% * False Acceptance Rate (FACC): 25.5% (This indicates a significant issue with accepting out-of-domain speech) * CPU Real-time Multiples (xCPUrt): 58.9

Probabilistic CFG Language Model (Trained on 485 utterances): * WER: 9.97% * FREJ: 5.57% * AWER: 4.6% * FACC: 9.09% (A substantial reduction in false acceptances) * xCPUrt: 19.0 (A significant speedup)

The results show that the Probabilistic CFG significantly reduces the False Acceptance Rate (from 25.5% to 9.09%) while also improving the Adjusted Word Error Rate and dramatically increasing recognition speed.

Limitations and Future Work

Limitations: * The paper acknowledges that the Probabilistic CFG still has a False Acceptance Rate of 9.09%, indicating room for further improvement in discriminating out-of-domain speech. * While the system is demonstrated, the paper focuses on the technical aspects of the dialogue interface rather than a full evaluation of its impact on geological exploration tasks.

Future Work: The authors suggest that the system has an "acceptably low false accept rate and is fast and accurate" with the probabilistic grammar. Future work would likely involve further refining the language model, potentially incorporating more advanced acoustic modeling, and integrating the dialogue system more deeply into the overall Mobile Agents project to evaluate its practical utility in simulated planetary exploration scenarios.

Summary updated

BibTeX
@inproceedings{dowding2003spoken,
  author = {John Dowding and James Hieronymus},
  title = {{A Spoken Dialogue Interface to a Geologist’s Field Assistant}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2003},
}
Figure