Spoken inquiry discrimination using bag-of-words for speech-oriented guidance system

Haruka Majima, Rafael Torres, Yoko Fujita, Hiromichi Kawanami, Tomoko Matsui, Hiroshi Saruwatari, Kiyohiro ShikanoInterspeech 2012

TL;DR

This paper tackles the challenge of distinguishing valid spoken queries from irrelevant noise (like background chatter or nonsensical utterances) in a real-world speech guidance system. By combining traditional acoustic features with "bag-of-words" representations derived from ASR results, they significantly improve discrimination accuracy, paving the way for more robust and user-friendly voice interfaces.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems are becoming ubiquitous, but their effectiveness in real-world environments is often hampered by unintended inputs. Imagine a voice-activated kiosk or a smart assistant: it's not enough for the system to understand what you say; it also needs to know if what you're saying is actually meant for it. This paper focuses on a speech-oriented guidance system called "Takemaru-kun," which provides information in a public space. A significant portion of user inputs are not actual queries but rather background conversations, laughter, coughs, or nonsensical utterances. These "invalid inputs" degrade the system's performance and user experience. Previous methods relied on acoustic features like Gaussian Mixture Model (GMM) likelihoods, but these struggled to differentiate between various types of invalid inputs, especially those with semantic content (like background conversations). The core problem is to build a system that can reliably filter out these irrelevant spoken inputs before they reach the main ASR processing pipeline.

Methods

The authors propose a novel approach to discriminate between valid and invalid spoken inquiries by leveraging multiple features beyond just acoustic information. Their method combines:

  • GMM Likelihood Values: This is the baseline approach, using GMMs trained on different categories of speech (adult valid, child valid, laughter, cough, noise, other invalid). High likelihood for invalid categories suggests rejection.
  • Bag-of-Words (BoW): This is a key innovation. They extract BoW representations from the 10-best ASR results. BoW essentially counts the frequency of words in a predefined vocabulary, capturing the semantic content of the utterance without needing a full understanding of grammar or syntax. This helps distinguish meaningful (even if invalid) utterances from pure noise.
  • Utterance Duration: The length of an utterance, determined by voice activity detection.
  • Signal-to-Noise Ratio (SNR): A measure of the signal quality relative to background noise.

To integrate these diverse features, they employ two powerful machine learning classifiers:

  • Support Vector Machine (SVM) with Radial Basis Function (RBF) kernel: SVMs are known for their effectiveness in high-dimensional spaces and finding optimal separating hyperplanes. They use a soft-margin approach to handle misclassifications.
  • Maximum Entropy (ME) method: ME models are flexible classifiers that can integrate various heterogeneous information sources, assigning weights to different features.

Datasets

The experiments were conducted using data collected from the "Takemaru-kun" system.

  • Training Data: One month of user utterances from August 2003.
  • Test Data: User utterances from October 2003.

The dataset was manually transcribed and labeled by four trained labelers, categorizing inputs into "valid inputs" and various types of "invalid inputs" (speech and non-speech). The paper notes that approximately 55.2% of all collected inputs were invalid.

Key Insights and Ideas

  • BoW is Crucial for Semantic Discrimination: The most significant insight is that incorporating Bag-of-Words (BoW) features, derived from ASR results, dramatically improves the system's ability to discriminate invalid inputs, especially those that are not pure noise but contain recognizable words (like background conversations). This suggests that understanding what is being said, even if it's not a direct query, is vital.
  • SVM Outperforms ME: While both SVM and ME classifiers showed improvements over the baseline, SVM consistently achieved higher F-measures across various feature combinations, indicating its superior performance for this specific task.
  • Combining Features is Key: The paper strongly advocates for a multi-feature approach. Simply relying on acoustic GMM likelihoods is insufficient. The synergistic effect of combining acoustic features with semantic (BoW) and signal quality (duration, SNR) features leads to the best results.
  • Real-World Data is Essential: The system's performance is evaluated on real user utterances collected in a public environment, highlighting the practical challenges and the necessity of robust discrimination methods for deployed systems.

Key Results

The paper reports significant improvements in the F-measure, a metric that balances precision and recall, for invalid input discrimination:

  • Baseline (GMM Likelihoods only): 81.7% F-measure.
  • SVM with GMM Likelihood + BoW: 85.0% F-measure (a 3.3% increase).
  • SVM with GMM Likelihood + BoW + Duration + SNR (all features): 86.6% F-measure. This represents a 4.9% improvement over the baseline.
  • ME with all features: Achieved 84.2% F-measure, also outperforming the baseline but less effectively than SVM.

The results demonstrate that the proposed multi-feature approach, particularly with SVM, significantly enhances the system's ability to filter out unwanted spoken inputs.

Limitations and Future Work

  • Limitations: The paper acknowledges that the BoW features are derived from the 10-best ASR candidates, implying that the quality of the ASR output directly impacts the BoW feature's effectiveness. The performance might be sensitive to ASR errors. The paper also mentions that the SVM classifier's parameters (like C) were set "a posteriori," suggesting a need for more automated or robust parameter tuning.
  • Future Work: The authors suggest further investigation into more effective combinations of different feature types. This could involve exploring more sophisticated feature extraction techniques or different machine learning models to further enhance discrimination performance.

Summary updated

BibTeX
@inproceedings{majima2012spoken,
  author = {Haruka Majima and Rafael Torres and Yoko Fujita and Hiromichi Kawanami and Tomoko Matsui and Hiroshi Saruwatari and Kiyohiro Shikano},
  title = {{Spoken inquiry discrimination using bag-of-words for speech-oriented guidance system}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure