Techniques for topic detection based processing in spoken dialog systems

Rajesh Balchandran, Leonid Rachevsky, Bhuvana Ramabhadran, Miroslav NovákInterspeech 2010

TL;DR

This paper tackles the challenge of making spoken dialog systems more natural by enabling them to directly understand user requests without requiring a separate application selection step. They propose and evaluate techniques for "topic detection" that allow systems to quickly identify the user's intent, offering a practical balance between accuracy, memory usage, and computational cost.

What Problem Does This Paper Address?

Modern spoken dialog systems often follow a rigid, multi-turn interaction pattern. For example, to play music, a user might first have to select the "music" application and then state their song request. This is inefficient and unnatural. The authors aim to enable "topic-selection-menu-free processing," where users can directly state their request (e.g., "Play 'Bohemian Rhapsody' by Queen") and the system can immediately understand the intent and topic. This requires robust topic detection, especially for dynamic content where the exact vocabulary isn't known beforehand (like an ever-growing music library). The core challenge lies in achieving high accuracy in topic detection while adhering to practical constraints of memory, computational cost, and the ability to easily add or remove topics from the system.

Methods

The paper explores and proposes several techniques for topic detection:

  • Global LM (Single Pass): A single language model (LM) is trained on data from all topics. The system performs one speech recognition pass using this model. While simple, it struggles with scalability and can lead to confusion between similar words across topics.
  • Two-Pass with Global LM: This approach first uses a Global LM for an initial recognition pass. Then, a classifier determines the topic, and the utterance is re-decoded using a topic-specific LM. This improves accuracy but inherits the Global LM's scalability issues.
  • Two-Pass with Parallel Language Models: This is a key proposed variant. Instead of a single Global LM, multiple topic-specific LMs are activated in parallel during the first pass. A graph factorization technique is used to reduce memory footprint. The system identifies the topic based on the best scoring LM. This is followed by a second pass using the identified topic-specific LM. This method offers better accuracy and flexibility.
  • Scaled-Down Language Models: To address memory constraints, the authors propose using "scaled-down" LMs. This involves techniques like using subsets of training data, pruning grammars for named entities (like song titles or addresses), or using phonetically balanced word sets. These scaled-down models can be applied to both Global LM and parallel LM approaches.

Datasets

The paper describes training data collected from volunteers and internal staff, covering various topics:

  • Music requests (song name, artist, album, genre) from a collection of 1000 songs.
  • Points of interest requests from 75 generic and 600 specific categories.
  • Name dialing requests from 250 phone contacts and 3 locations.
  • Addresses from across the United States.

The test data consisted of:

  • 1576 music requests
  • 2099 name dialing requests
  • 470 requests for points of interest

The experiments used IBM's Embedded Via Voice (EVV) for speech recognition and action classification.

Key Insights and Ideas

  • The Trade-off is Real: The paper highlights a fundamental tension between accuracy, memory, and computational cost in topic detection. Improving one often negatively impacts another, necessitating a careful balance.
  • Parallel LMs Shine: The proposed "Two-pass with Parallel Language Models" approach (T31) emerges as a strong contender, offering high topic detection accuracy and good ASR performance, closely approaching the ideal of using isolated topic-specific LMs.
  • Memory Matters: When memory is a significant constraint, using scaled-down language models (T32) provides a viable compromise, maintaining reasonable accuracy while reducing memory footprint.
  • Topic Detection Accuracy is Crucial: The experiments show a strong correlation between topic detection accuracy and the final ASR word error rate. Incorrect topic identification directly leads to poorer recognition.

Key Results

  • T31 (Parallel LMs, Full Data) achieved the highest topic detection accuracy across all tested topics (Music, Name Dialing, Points of Interest), ranging from 93.40% to 98.66%. It also yielded the lowest ASR word error rates, close to the ideal of isolated topic-specific LMs.
  • T32 (Parallel LMs, Scaled Down) offered a good balance when memory was an issue, with topic detection accuracies between 90.85% and 96.00% and ASR word error rates around 7.88% to 21.69%.
  • The Global LM approaches (T21, T22) performed less effectively, especially in terms of topic detection accuracy, with T21 achieving 74.04% for Points of Interest and T22 achieving 70.65% for Name Dialing.
  • The paper concludes that T31 is preferable when memory is not a concern, while T32 is a suitable alternative when memory is limited.

Limitations and Future Work

  • Scalability of Global LM: The paper notes that the Global LM approach, while simple, does not scale well as more topics are added, requiring the LM to be rebuilt and potentially increasing confusion.
  • Memory Constraints: The performance of scaled-down LMs is generally not as good as their full-data counterparts, indicating that memory limitations do impose a performance penalty.
  • Future Work: The authors suggest that the parallel LM approach offers a promising direction for building more natural and efficient spoken dialog systems. Further research could explore more advanced techniques for scaling and optimizing these parallel models.

Summary updated

BibTeX
@inproceedings{balchandran2010techniques,
  author = {Rajesh Balchandran and Leonid Rachevsky and Bhuvana Ramabhadran and Miroslav Novák},
  title = {{Techniques for topic detection based processing in spoken dialog systems}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure