TL;DR
This paper introduces a discriminative training approach for multinomial topic classifiers used in speech routing. By optimizing a novel objective function based on n-norm posterior probabilities, they significantly reduce classification errors compared to traditional maximum likelihood methods, achieving up to a 45% reduction in an NLU task.
What Problem Does This Paper Address?
Speech routing, a crucial component in systems like call centers and NLU applications, involves classifying an utterance into a specific topic. Traditional methods often rely on Maximum Likelihood (ML) estimation, which can lead to higher error rates because they only consider data from the true class. This paper addresses the limitation of ML by proposing a discriminative training (DT) method that leverages information from all competing classes, aiming to improve classification accuracy and robustness. The challenge of feature sparsity, especially when using n-grams, is also tackled.
Methods
The core contribution is a discriminative training (DT) method for multinomial topic classifiers. Instead of maximizing the likelihood of the correct class, the DT approach maximizes an n-norm posterior probability of the correct topic. This objective function is designed to better handle the competition between different topics.
Key methodological aspects include:
- Objective Function: The paper defines an objective function based on the n-norm of the posterior probability of the true topic. This allows for flexibility, with n=1 corresponding to the standard posterior probability and n=infinity focusing on the highest joint probability.
- Parameterization: To avoid issues with negative probability estimates during training, they propose an alternative parameterization using exponents (a_w|T) instead of direct probabilities (P_w|T). This ensures probabilities remain positive.
- Gradient-Based Optimization: The objective function is maximized using gradient ascent, requiring the computation of derivatives with respect to model parameters.
- N-gram Feature Selection: To combat sparsity and improve efficiency when using n-grams, they employ a "stickiness" measure (based on Mutual Information) to select the most discriminative n-grams.
- Explicit Word Pruning: To further alleviate sparsity and prevent over-training, they propose methods to prune noisy or non-discriminative words using Mutual Information or entropy measures.
- Post-processing: The authors explore using Decision Trees (DT) and Multi-Layer Perceptrons (MLP) to post-process the classification results, aiming to correct errors from the primary classifier.
Datasets
The experiments were conducted using three datasets associated with a financial retirement planning telephony application. These datasets varied in vocabulary size, number of topics, and corpus size. Specific details about the datasets (perplexity, vocabulary size, number of training sentences, and number of topics) are provided in Table 1. No new benchmarks were introduced; existing datasets were used for evaluation.
Key Insights and Ideas
- Discriminative Training Outperforms ML: The fundamental insight is that discriminative training, by considering competing classes, significantly improves topic classification accuracy over traditional Maximum Likelihood methods.
- N-norm Objective Function: The flexibility of the n-norm objective function allows for tuning the classifier's behavior, potentially focusing more on the top-ranked topics.
- Soft-Pruning via Exponents: Using exponents for parameterization naturally handles probability constraints and provides a "soft-pruning" mechanism where words with zero exponents effectively disappear from the model.
- Feature Engineering is Crucial: Techniques like n-gram selection (using "stickiness") and explicit word pruning are vital for managing feature sparsity and improving model performance and robustness.
- Post-processing Adds Value: Decision Tree models, in particular, can further refine classification results, suggesting that combining multiple models or using them for error correction is a promising avenue.
- DT Robustness: Discriminative training appears robust, maintaining comparable performance even when a significant portion of the vocabulary is pruned, indicating redundancy in word information and the power of DT.
Key Results
- The proposed discriminative trained multinomial system (DT-MN) reduced the classification error rate by up to 45% compared to the baseline non-discriminative multinomial system (ML-MN) on an NLU task of financial transactions.
- On Set 1, the best DT configuration achieved an error rate of 4.53%, compared to 8.18% for the ML-MN baseline.
- An n-best oracle experiment on test set 2 showed that if the true topic was among the top-3 highest probability topics, the classification error rate dropped from 19.7% to 5.7% for the DT-MN system, highlighting the quality of its ranked outputs.
- Post-processing with a Decision Tree Model (DTM) reduced the classification error rate in Set 2 from 19.8% to 18.7%.
- Using n-grams (4-gram) with ML-MN improved performance over unigram ML-MN, but n-gram DT-MN was slightly worse than unigram DT-MN on Set 3, possibly due to over-fitting with a larger feature set.
- Pruning words using Mutual Information showed that performance remained comparable even after pruning 1500 words, demonstrating the robustness of the DT-MN system.
Limitations and Future Work
- Over-fitting with N-grams: The paper notes that n-gram models, especially with discriminative training, can be prone to over-fitting, particularly on datasets with many frequently appearing utterances.
- MLP Post-processing Limitations: While DTM showed improvement, MLP post-processing combined with majority voting did not yield substantial benefits, though it did reveal complementary strengths between MLP and DT-MN.
- Future Work: The authors suggest that developing more judicious combination systems to exploit off-diagonal components in confusion matrices (where different models disagree but one is correct) could lead to better hypothesis combination and further error reduction. They also imply that further exploration of n-gram configurations and pruning strategies could be beneficial.
Summary updated