A hybrid SVM/MCE training approach for vector space topic identification of spoken audio recordings

Timothy J. Hazen, Fred RichardsonInterspeech 2008

TL;DR

This paper introduces a novel hybrid training approach that combines Support Vector Machines (SVMs) with Minimum Classification Error (MCE) training to improve topic identification in spoken audio. By using MCE to learn optimal feature weighting for SVMs, the authors achieve significant reductions in topic identification error rates, outperforming previous methods.

What Problem Does This Paper Address?

Topic identification (topic ID) aims to determine the subject matter of audio or text documents. Support Vector Machines (SVMs) are a popular and effective tool for this task, but their performance heavily relies on how the input features are normalized. In spoken audio, especially when using Automatic Speech Recognition (ASR) outputs, common but uninformative words (like "the" or "a") can dominate the feature space, obscuring the signal from rarer, more informative content words. This can lead to suboptimal performance for SVM classifiers. The authors tackle this challenge by developing a method to learn an optimal feature normalization (weighting) specifically for SVMs, aiming to boost their accuracy in topic ID tasks.

Methods

The core innovation lies in a hybrid training strategy that leverages MCE for feature weighting within an SVM framework.

  • Vector Space Model: Both Naive Bayes (NB) and SVM classifiers can be viewed as operating in a vector space. Documents are represented as vectors of word counts (or relative frequencies), and classifiers learn a "routing matrix" to map these vectors to topic scores.
  • SVM with Linear Kernel: The paper focuses on linear SVMs, where the kernel function is a simple dot product. A key challenge here is that raw word frequencies can be dominated by common words.
  • Term Weighting: To address this, the authors adapt a term weighting vector ($\phi$) that modifies the input feature vectors before they are used in the SVM's linear kernel. This is akin to applying a normalization scheme.
  • MCE Training for Feature Weighting: The novel part is using Minimum Classification Error (MCE) training to learn the optimal values for this term weighting vector ($\phi$). MCE training directly optimizes a misclassification measure, aiming to minimize the probability of making errors. This is applied to learn the weights ($\lambda_w$) for each feature (word).
  • Hybrid Training: The process involves iteratively:
  • Training an SVM with an initial feature weighting.
  • Using MCE to train a new feature weighting vector based on the SVM's performance.
  • Retraining the SVM with the newly learned feature weights. This cycle is repeated to refine both the feature weights and the SVM model.

  • Back-End Calibration: For improved performance, a regression-based "back-end" classifier (FoCal toolkit) is used to calibrate the output scores from the primary topic ID classifier.

Datasets

The experiments were conducted on the English Phase 1 portion of the Fisher Corpus. This corpus consists of 5851 recorded telephone conversations covering 40 different topics. The data was split into:

  • Recognizer training set
  • Topic ID training set
  • Topic ID development test set (used for parameter tuning)
  • Topic ID evaluation test set (used for final results)

The evaluation was performed on both original text transcripts and outputs from an Automatic Speech Recognition (ASR) system.

Key Insights and Ideas

  • MCE for Feature Weighting is Generalizable: The paper demonstrates that an MCE-based feature weighting approach, previously successful with Naive Bayes, can be effectively adapted to linear SVMs due to their shared vector space representation.
  • Feature Weighting is Crucial for SVMs in Topic ID: Raw word frequencies are problematic for SVMs in topic ID. Learning optimal feature weights significantly improves performance by giving more importance to discriminative words.
  • Hybrid Training Yields Synergistic Gains: Combining MCE for feature weighting with SVM training leads to better results than either method alone. The MCE process refines the feature space, allowing the SVM to learn a more robust decision boundary.
  • MCE Improves Calibration: The authors observed that MCE training appears to produce scores that are better calibrated, reducing the need for extensive back-end calibration.

Key Results

  • Significant Error Rate Reduction: The hybrid SVM/MCE approach achieved substantial improvements. For example, on the ASR output, the SVM error rate was reduced from 11.2% (initial SVM) to 7.7% (SVM + MCE + BE).
  • Competitive Performance: The hybrid SVM/MCE system achieved performance comparable to, and in some cases better than, their previously developed Naive Bayes systems trained with MCE.
  • Back-End Calibration Benefits: While MCE training improved scores, back-end calibration still provided additional gains, particularly for the initial SVM system.

Limitations and Future Work

  • Over-tuning: The authors noted that excessive iterations of the hybrid SVM/MCE training could lead to over-tuning to the training set, harming performance on the development set.
  • MCE vs. SVM Objectives: The paper acknowledges that the MCE training objective (minimizing average loss) is different from the SVM objective (maximizing margin), and there's no guarantee that MCE improvements will always translate to better SVM performance after retraining.
  • Further Investigation on Calibration: The observation that MCE training improves score calibration warrants further investigation.
  • Exploring Other Kernels: While the paper focused on linear kernels, exploring the hybrid approach with non-linear SVM kernels could be a future direction.

Summary updated

BibTeX
@inproceedings{hazen2008hybrid,
  author = {Timothy J. Hazen and Fred Richardson},
  title = {{A hybrid SVM/MCE training approach for vector space topic identification of spoken audio recordings}},
  booktitle = {Interspeech},
  year = {2008},
}
Figure