TL;DR
This paper introduces a novel method for classifying spoken utterances that bypasses the need for manual transcription of training data. By combining unsupervised training of acoustic models with off-the-shelf classifiers, they achieve classification accuracy surprisingly close to traditional methods that rely on expensive human transcriptions. This significantly reduces the effort required to build new spoken language applications.
What Problem Does This Paper Address?
Building data-driven speech processing applications, such as voice assistants or automated customer service systems, often requires a massive amount of transcribed training data. Manually transcribing spoken utterances into text is a time-consuming and costly bottleneck. This is particularly problematic for tasks like utterance classification, where the goal is to assign a specific meaning or "action" to a spoken phrase (e.g., routing a call to the correct department). Existing methods for utterance classification typically rely on word-based language models, which necessitate these expensive transcriptions. The paper aims to overcome this limitation by developing a method that can learn to classify utterances effectively without requiring any human-annotated text.
Methods
The proposed method, dubbed "PhonesMM," has two main phases: training and runtime operation.
Training: 1. Unsupervised Phonotactic Model Training: The core innovation lies here. Instead of learning word sequences, the system learns phone sequences. A phone n-gram model is trained using only the audio of the training utterances and a pre-existing, domain-independent acoustic model. This training is an iterative process where the output phone strings from the current model are used to refine the n-gram model for the next iteration. This is akin to a "Viterbi re-estimation" or "1-best re-estimation" process. 2. Phone String Classifier Training: Once the phone n-gram model is trained, it's used to generate phone strings for the training utterances. These phone strings, along with contextual information (like the prompt that elicited the utterance), are then used to train a classification model (specifically, BoosTexter, an AdaBoost-based classifier). This model learns to map phone strings and prompts to the desired actions.
Runtime Operation: At runtime, a new utterance is first converted into a phone string using the trained phonotactic model and the acoustic model. This phone string is then fed into the trained classifier, which predicts the corresponding action and a confidence score.
A key detail in the training is how to adjust recognition parameters. To ensure the recognized phone strings are representative of natural speech, the "word insertion penalty" for the recognizer is adjusted so that the speaking rate (in phones per second) of the automatically detected speech matches an estimated speaking rate for the domain.
Datasets
The experiments were conducted on three distinct collections of utterances from different domains:
- Domain A: Customer-side live English conversations with an automated customer care system. This is a large dataset with 40,106 training and 9,724 test utterances, averaging 11.29 words per utterance. It had 56 actions.
- Domain B: Utterances from an interactive spoken language application for product line information. This dataset had 10,470 training and 5,005 test utterances, with an average length of 3.95 words. It had 54 actions.
- Domain C: Utterances from an interactive spoken language application for consumer order transactions. This dataset contained 14,355 training and 5,000 test utterances, averaging 8.88 words. It had 93 actions.
These datasets are described as being from "similar interactive spoken natural language systems," with Domain A being a previously studied benchmark.
Key Insights and Ideas
- Unsupervised Learning for Utterance Classification is Viable: The most significant insight is that effective utterance classification can be achieved without manual transcriptions, a long-standing bottleneck. This opens the door to much faster and cheaper development of spoken language applications.
- Phone Sequences as a Proxy for Meaning: The paper demonstrates that sequences of phones, learned unsupervisedly, can capture enough information about the utterance's intent to rival word-based methods. This suggests that the acoustic realization of speech carries strong semantic cues.
- Domain-Independent Acoustic Models are Sufficient: The method relies on a general-purpose acoustic model, not one specifically trained for each domain. This further reduces the setup effort.
- BoosTexter's Adaptability: The use of BoosTexter, a powerful text classification algorithm, highlights its ability to learn from diverse features, including phone n-grams and prompts, to achieve high accuracy.
- Implicit Word Boundary Modeling: While the method operates on phones, the unsupervised re-estimation of phone n-grams means that two-word units can be implicitly modeled, as phone sequences can naturally straddle word boundaries.
Key Results
The core finding is the strong performance of the PhonesMM method compared to traditional word-based methods (WordsHM and WordsHH) that require manual transcriptions.
- High Accuracy: At 0% rejection rate, PhonesMM achieved accuracies of 74.6% (Domain A), 68.2% (Domain B), and 73.3% (Domain C). These are remarkably close to the word-based methods, which achieved 76.2%/77.0% (Domain A), 68.9%/69.9% (Domain B), and 73.7%/74.9% (Domain C).
- Small Performance Gap: The difference in accuracy between the unsupervised phone-based method and the supervised word-based methods was consistently small, typically around 1-2% absolute, and even less than 1% when high rejection rates were acceptable.
- Effectiveness of Unsupervised Training: Tables 5, 6, and 7 show that increasing the unsupervised training iteration (Nmax) significantly improved both phone recognition accuracy and classification accuracy across all domains, demonstrating the value of the iterative refinement.
- Runtime Efficiency: For all conditions, the median recognition and classification time was less than real-time.
Limitations and Future Work
- Slightly Lower Accuracy: While close, the phone-based method is consistently slightly lower in accuracy than the word-based methods when transcriptions are available. The authors suggest that if transcriptions are already available, they can be used for a slight performance boost.
- Domain Complexity: The paper notes that domain complexity (utterance length, number of actions) influences classification rates. Domain B, with shorter utterances, performed better, while Domain C, with more actions, performed lower.
- Future Work: The paper doesn't explicitly detail future work, but the success of this method suggests exploring more sophisticated unsupervised learning techniques for acoustic models, investigating different classifier architectures, and applying this approach to a wider range of spoken language tasks. The potential for further reducing the effort in building new applications remains a strong motivator.
Summary updated