A dynamic vocabulary spoken dialogue interface

Stephanie Seneff, Chao Wang, Lee Hetherington, Grace ChungInterspeech 2004

TL;DR

This paper introduces a novel spoken dialogue interface that dynamically updates its vocabulary and language models on-the-fly, allowing it to adapt to changing information and user queries without requiring full system recompilation. This approach significantly improves recognition performance by reducing the vocabulary size at any given time, making it more efficient and robust for domains with evolving data.

What Problem Does This Paper Address?

Traditional spoken dialogue systems are limited by fixed vocabularies and grammars that must be pre-defined before runtime. This rigidity poses a significant challenge when dealing with domains where information changes frequently, such as restaurant databases or online catalogs. Updating these systems requires recompiling the entire linguistic structure, which is time-consuming and impractical. The authors tackle this problem by proposing a "dynamic vocabulary" approach, enabling the system to incorporate new words and entities into its recognition and natural language understanding components in real-time, as the dialogue context evolves. This allows for a more flexible and adaptive dialogue experience, especially in domains with a large and changing set of entities.

Methods

The core innovation lies in a dynamic vocabulary capability integrated into both the speech recognizer and the natural language (NL) grammar.

  • Speech Recognizer: The system uses a speech recognizer that can dynamically update its Finite State Transducers (FSTs). Specifically, certain word classes (like restaurant names) are designated as "dynamic." These dynamic classes are updated before each user turn. An "out-of-vocabulary" (OOV) word model is also employed to handle words not present in the current dynamic vocabulary.
  • NL Understanding: The NL parser (TINA) is augmented to handle sequences within special markers (<dynrestaurant> and </dynrestaurant>) that are added by the recognizer for dynamic vocabulary items. A generic "any_sequence" category is used to parse these dynamic sequences, which are then mapped to semantic categories by the dialogue manager.
  • Dialogue Management and Dynamic FST Creation: A specialized "dynamic FST Galaxy server" is implemented. The dialogue manager triggers this server to create and update dynamic grammar FSTs based on the current dialogue context. This involves retrieving new restaurant names and their pronunciations (or generating them using a letter-to-sound module) and compiling them into a dynamic FST that is then uploaded to the recognizer.
  • Two-Pass Recognition: For queries involving previously unknown restaurant names, a two-pass recognition strategy is used. The first pass identifies the unknown word. The dialogue manager then uses context from the utterance to filter the database and update the dynamic vocabulary. The second pass re-recognizes the utterance with the newly updated dynamic vocabulary, aiming to resolve the unknown word.

Datasets

The paper primarily uses simulated dialogues for training and evaluation.

  • Training: Thousands of simulated dialogues were generated for debugging and for training the recognizer's language models and the NL grammar.
  • Evaluation: Experiments were conducted using simulated data. The paper mentions a restaurant information domain for Boston, containing 982 restaurants across 106 cities and 45 neighborhoods, with detailed information for each. This data was used to populate the system's database and generate simulated queries.

Key Insights and Ideas

  • Vocabulary Size Matters: Dynamically reducing the vocabulary size for specific tasks significantly improves recognition performance by reducing lexical confusion.
  • Real-time Adaptability: The ability to update vocabulary and grammars on-the-fly, without full system recompilation, is crucial for dialogue systems operating in dynamic environments.
  • Integration is Key: The success of dynamic vocabulary relies on tight integration between the speech recognizer, NL parser, and dialogue manager.
  • Two-Pass for Unknowns: A two-pass recognition strategy can effectively handle queries for new entities by first identifying them as unknown and then re-recognizing with an updated vocabulary.

Key Results

  • Experiment I (Prior Context): For a set of 315 utterances, the dynamic vocabulary system (System 2) achieved a Word Error Rate (WER) of 9.5% compared to 12.6% for a full vocabulary system (System 1). For sentences containing proper nouns, the WER was reduced from 14.3% to 7.5%.
  • Experiment II (No Prior Context): For 197 utterances, the dynamic system showed a degradation in WER (14.3% vs. 12.0% for System 1). However, the paper notes that task success rates were comparable, suggesting that while recognition might be slightly worse, the overall goal completion could still be achieved. The authors attribute this degradation to challenges in correctly identifying OOV words and resolving names in the two-pass procedure.

Limitations and Future Work

  • Limitations:
    • The current system relies heavily on simulations for development and testing; real user data is needed for more robust evaluation.
  • Experiment II showed a degradation in WER, indicating challenges in the two-pass procedure for handling utterances without prior context.
  • The system currently prompts the user for more context when an unknown word cannot be resolved, rather than employing more sophisticated error recovery subdialogues.
  • The current implementation uses a fixed database, although the framework supports a dynamically decoupled database.
  • Future Work:
    • Conduct full-scale evaluations with real user data.
  • Expand the dynamic vocabulary capability with clarification subdialogues to help users repeat or spell unknown words.
  • Investigate methods for automatic generation of restaurant aliases.
  • Explore supporting dynamically decoupled databases and searching the Web in real-time for data.
  • Compare systems trained on real user data versus simulated data.

Summary updated

BibTeX
@inproceedings{seneff2004dynamic,
  author = {Stephanie Seneff and Chao Wang and Lee Hetherington and Grace Chung},
  title = {{A dynamic vocabulary spoken dialogue interface}},
  booktitle = {Interspeech},
  year = {2004},
}
Figure