Automatic Acquisition of Names Using Speak and Spell Mode in Spoken Dialogue Systems

Grace Chung, Stephanie Seneff, Chao WangNAACL 2003

TL;DR

This paper introduces a novel "speak and spell" mode for spoken dialogue systems to automatically acquire unknown names. It uses a multi-stage recognition process that first extracts letter hypotheses from the spelling part of an utterance and then combines this with the spoken name to deduce both the spelling and pronunciation, enabling real-time system updates.

What Problem Does This Paper Address?

Spoken dialogue systems are becoming more sophisticated, but their knowledge bases are often static, requiring manual updates by developers. A significant enhancement would be the ability for these systems to automatically learn new information, such as unknown words, directly from user interactions. This paper focuses on the challenging task of acquiring new names, which are particularly difficult due to English's irregular letter-to-sound rules and the diverse linguistic origins of names. The authors propose a "speak and spell" mode where users say their name and then spell it sequentially, aiming for a natural dialogue flow while overcoming the technical hurdles of boundary detection between spoken and spelled segments and integrating this new knowledge seamlessly into the system.

Methods

The core of the proposed solution is a multi-stage recognition procedure designed to handle utterances containing both a spoken name and its spelling.

  1. First Stage: Letter Recognition: This stage uses a segment-based recognizer with an "unknown word" model. It processes the spelled portion of the utterance to extract letter hypotheses. This is crucial because it treats the spoken name as a generic out-of-vocabulary (OOV) word and focuses on the more structured spelling input. The output is a lattice (FST) containing an unknown word label and letter hypotheses.
  2. Intermediate Stage: FST Composition: This stage takes the output from the first stage and constructs a reduced search space for the subsequent recognition pass. It involves a series of Finite State Transducer (FST) compositions. Key components include:
  3. A subword trigram language model to prune improbable sequences.
  4. The ANGIE (Acoustic-Phonetic-Graphemic) framework, a hierarchical model that encodes subword structure and predicts grapheme-phoneme mappings. ANGIE is used to generalize from observed data to unseen words.
  5. Phonological rules are applied to expand phoneme units into allowable phonetic sequences.
  6. This stage creates two main FSTs: F1 (for the spoken part, incorporating letter hypotheses and ANGIE mappings) and F2 (for the spelled part).
  7. Final Stage: Sound-to-Letter Recognition: This stage performs a new search using the enriched language models and the reduced search space defined by the concatenated F1 and F2 FSTs. It integrates acoustic information from both the spoken and spelled parts of the waveform, along with language model information from ANGIE, to generate the final spelling and phonemic sequences for the name.
  8. Phoneme Extraction: A separate pass is used to extract the phoneme sequence, typically performed after the user confirms the spelling. This process is similar to the sound-to-letter recognition but uses the confirmed letter sequence to constrain the search.

The system is integrated into the ORION spoken dialogue system using a multi-threaded architecture to ensure real-time operation, allowing the main dialogue to continue while name acquisition is processed in parallel.

Datasets

  • Training Data: The ANGIE framework was trained on a lexicon of proper nouns, initially containing over 75,000 entries, with an additional 25,000 names added automatically.
  • Evaluation Data:
    • Test Set A: Contains 416 words that are present in the ANGIE 100K training vocabulary.
  • Test Set B: Contains 219 words that are previously unseen (OOV) in the training data.
  • The test sets were compiled from a preliminary ORION data collection and utterances from the JUPITER system, where users were asked to speak and spell their names.

Key Insights and Ideas

  • "Speak and Spell" as a Natural Interaction: The authors posit that this mode is the most intuitive way for users to provide their names, balancing naturalness with the need for explicit spelling.
  • Leveraging ANGIE for OOV Generalization: The ANGIE framework is central to handling OOV names by encoding subword structure and enabling generalization from observed grapheme-phoneme mappings.
  • Multi-Stage FST Processing for Constrained Search: The paper demonstrates the power of FST composition to progressively build a highly constrained search space, integrating various knowledge sources (acoustic, language models, ANGIE) for efficient and accurate recognition.
  • Handling Boundary Detection: The multi-stage approach implicitly addresses the challenge of identifying the transition between the spoken name and its spelling by treating them as distinct segments with different recognition strategies.
  • Real-time System Integration: The use of parallel threads and asynchronous system updates is crucial for maintaining a smooth user experience while incorporating new knowledge.

Key Results

  • Significant WER Reduction: The multi-stage system dramatically improves Word Error Rate (WER) compared to a single-stage letter recognizer.
  • For Test Set A (in-vocabulary), WER dropped from 40.4% (Stage 1) to 25.7% (Multi-Stage System I).
  • For Test Set B (OOV), WER dropped from 58.9% (Stage 1) to 48.9% (Multi-Stage System I).
  • Phoneme Error Rates (PER): PER was evaluated on both correct and incorrect letter hypotheses. For Test Set A, PER was 15.0% (overall), and for Test Set B, it was 25.5%.
  • Impact of Subword Trigram: Removing the subword trigram improved performance on OOV words (Test Set B) but slightly degraded it for in-vocabulary words (Test Set A), suggesting that OOV words benefit from a less constrained language model.

Limitations and Future Work

  • Confusable Letter/Phoneme Pairs: The paper notes that acoustically similar letter pairs (e.g., M/N, A/E) and phoneme pairs (e.g., vowels, m/n) contribute to errors.
  • Transition Detection: Errors in detecting the transition from the spoken name to the spelling part are acknowledged as a source of deletions and insertions.
  • Future Work:
    • Prosodic Features: Incorporating prosodic features could improve the detection of the spelling onset.
  • Stricter Search Constraints: Explicitly enforcing agreement between spoken and spelled parts during search, or using length constraints when hypotheses are inconsistent, could enhance accuracy.
  • Parallel In-Vocabulary Modeling: Experimenting with a parallel component that explicitly models common in-vocabulary words could boost overall accuracy by reducing reliance on the ANGIE stage for frequently encountered names.
  • Single Recognition Server: Consolidating the multi-stage processing into a single recognition server for potentially faster real-time operation.
  • General OOV Acquisition: Extending the methodology to learn pronunciations and spellings for broader classes of OOV words, such as geographical locations or businesses.

Summary updated

BibTeX
@inproceedings{chung2003automatic,
  author = {Grace Chung and Stephanie Seneff and Chao Wang},
  title = {{Automatic Acquisition of Names Using Speak and Spell Mode in Spoken Dialogue Systems}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2003},
}
Figure