From Speech Recognition to Spoken Language Understanding

Victor Zue, James R. Glass, David Goodine, Lynette Hirschman, Hong C. Leung, Michael S. Phillips, Joseph Polifroni, Stephanie Seneff

TL;DR

This paper introduces the VOYAGER system, an urban navigation and exploration tool that moves beyond basic speech recognition to achieve spoken language understanding. It highlights the crucial integration of speech recognition (SUMMIT) and natural language processing (TINA) with a control strategy to enable interactive, context-aware spoken dialogue for practical applications.

What Problem Does This Paper Address?

While speech recognition technology has advanced significantly, enabling efficient speech transmission and generation, its practical application for human-computer interaction remains limited. Current systems often struggle with continuous speech, unrestricted vocabularies, and adverse acoustic conditions due to an incomplete understanding of how linguistic information is encoded in speech. This paper addresses the challenge of bridging the gap between recognizing spoken words and truly understanding the meaning of spoken language, enabling more natural and effective human-machine communication. The authors focus on developing systems that can handle realistic, interactive problem-solving scenarios, including spontaneous speech with hesitations and false starts.

Methods

The VOYAGER system is built upon three core components:

  • SUMMIT Speech Recognition System: This system converts the speech signal into a set of word hypotheses. It begins by transforming the speech signal using an auditory model, identifying acoustic landmarks, and mapping these to phoneme hypotheses. Phonetic classification is enhanced using multi-layer perceptrons. Lexicon representation uses pronunciation networks generated by phonological rules, and Viterbi decoding finds the best path matching acoustic-phonetic and lexical networks.
  • TINA Natural Language System: TINA is designed to both provide constraints to the speech recognizer and interpret the meaning of the utterance for the back-end. It's a probabilistic parser that operates top-down with an agenda-based control strategy. TINA uses a shared network structure derived from context-free rules, augmented with syntactic and semantic constraint filters. Probabilities are acquired automatically from training data and are pooled across arcs to create a hierarchical bigram model.
  • Control Strategy: An N-best algorithm (Chow & Schwartz, 1989; Soong & Huang, 1990) mediates between SUMMIT and TINA. The recognizer proposes its top N sentence hypotheses, and TINA acts as a filter, accepting the first hypothesis it can successfully parse. If parsing fails, TINA can request clarification or provide a rejection message. A word-pair language model is used to constrain theories.

Datasets

The system was trained and tested using a corpus of spontaneous speech.

  • Training Set: Data from 70 speakers (equally divided between male and female).
  • Development Set: Data from 20 additional speakers.
  • Test Set: Data from 10 subjects (5 male, 5 female), consisting of 485 utterances. The average number of words per sentence was 7.7.

Key Insights and Ideas

  • Integration is Key: The paper strongly emphasizes that moving beyond speech recognition to spoken language understanding requires a tight integration of speech recognition, natural language processing, and a control strategy. TINA's role as a filter for SUMMIT's hypotheses is a prime example.
  • Realistic Domains Drive Research: Using a realistic application domain like urban navigation (VOYAGER) is crucial for collecting data on how people actually use spoken language to access information and solve problems, and for rigorous evaluation.
  • Context is King for Understanding: The VOYAGER system demonstrates the importance of discourse capabilities, including handling pronominal reference and underspecified queries, to make spoken interactions feel natural and effective.
  • Spontaneous Speech is a Major Hurdle: The authors acknowledge that spontaneous speech phenomena (hesitations, false starts) are significant contributors to recognition errors and parsing failures.

Key Results

  • VOYAGER Action Generation: The system successfully generated an action for 51.7% of the sentences in the test set.
  • Parsing Failures: 48.3% of sentences failed to parse, attributed to:
  • Recognition errors or inability to handle spontaneous speech (20.3%)
    • Unknown words (17.2%)
    • Linguistic structures that would not have parsed even if recognized correctly (10.5%)
  • Response Accuracy: When a parse was generated, the system almost never failed to provide a response. Of the sentences that did generate an action, only 13% resulted in an incorrect response. The remainder received the "I'm sorry but I didn't understand you" message.
  • Back-end Evaluation: Naive users judged about 80% of the back-end responses to be appropriate, with an additional 5% being verbose but correct.
  • Processing Speed: The system processed utterances in 3 to 5 times real-time on a SUN-4 workstation.

Limitations and Future Work

  • Limitations:
    • The system's performance is significantly impacted by recognition errors and the challenges of spontaneous speech.
  • The current discourse capabilities are described as "simplistic."
    • A substantial portion of utterances failed to parse, even if recognized correctly, indicating limitations in the natural language understanding component or its integration with the back-end.
  • Future Work:
    • Improving phonetic recognition accuracy of SUMMIT by incorporating context-dependent models.
  • Investigating control strategies that more fully integrate speech recognition and natural language components.
  • Expanding the system's ability to handle more complex interactions and a wider range of linguistic phenomena.

Summary updated

Abstract

Spoken language is one of the most natural, efficient, flexible, and econom(cid:173) ical means of communication among humans. As computers play an ever increasing role in our lives, it is important that we address the issue of providing a graceful human-machine interface through spoken language. In this paper, we will describe our recent efforts in moving beyond the scope of speech recognition into the realm of spoken-language understand(cid:173) ing. Specifically, we report on the development of an urban navigation and exploration system called VOYAGER, an application which we have used as a basis for performing research in spoken-language understanding.
BibTeX
@inproceedings{zue1990from,
  author = {Victor Zue and James R. Glass and David Goodine and Lynette Hirschman and Hong C. Leung and Michael S. Phillips and Joseph Polifroni and Stephanie Seneff},
  title = {{From Speech Recognition to Spoken Language Understanding}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1990},
}
Figure