TL;DR
This paper explores using sub-word units (like syllables) instead of full words to represent spoken German documents for classification. They found that this approach, especially when combined with Linear Support Vector Machines (LSVMs), significantly improves classification accuracy, even with imperfect speech recognition. This suggests a promising avenue for building more robust spoken document retrieval systems.
What Problem Does This Paper Address?
Spoken document classification, a key component of spoken document retrieval systems, faces a significant challenge: the inherent error rate of Automatic Speech Recognition (ASR) systems. When ASR systems misrecognize words, these errors can propagate and degrade the performance of downstream classification tasks. Traditional approaches often rely on word-level features, but these can be lost due to ASR errors, especially in systems designed for broad applicability (speaker and domain independent) rather than specialized tasks.
The authors identify two main drawbacks to using sub-word units:
- Loss of Discriminative Power: Directly using sub-word units might lose the discriminative information present at the word level.
- Increased Feature Space Complexity: Expanding sub-word units into higher-level features (like combinations) to compensate for lost word-level information can lead to a massive increase in the feature space, risking overfitting.
This paper investigates whether the benefits of sub-word features can outweigh these drawbacks, particularly in the context of a challenging, real-world classification task.
Methods
The core of the paper lies in comparing different feature representations and classification algorithms.
- Feature Extraction: Instead of relying solely on recognized words, the system utilizes sub-word units, specifically syllables. The ASR system uses a syllable-based language model trained on a large corpus of German parliamentary proceedings. This model is designed to be domain-independent. The recognized syllable sequences are then expanded into various feature representations:
- Syllable n-grams: Combinations of consecutive syllables (e.g., syllable bigrams).
- Phoneme n-grams: Syllables are further broken down into phonemes, and these phoneme sequences are used to create n-grams. This approach is inspired by string kernels used in bioinformatics.
- Classifiers: Two classifiers are compared:
- Naive Bayes: A simple, well-established probabilistic classifier. It's used as a baseline and for comparison.
- Linear Support Vector Machine (LSVM): A powerful classifier known for its ability to handle high-dimensional feature spaces and control complexity independently of dimensionality. The authors specifically use linear kernels, focusing on the LSVM's capacity for complexity control.
- Training Strategy: Experiments were conducted using both text transcripts and ASR output. A key experiment also explored training classifiers on text and then applying them to ASR output, aiming to reduce the need for audio training data.
Datasets
- Deutsche Welle Kalenderblatt Database: This is the primary dataset used for evaluation. It consists of 841 German radio programs (approximately 5 minutes each) with accompanying transcripts. The programs cover a wide range of topics and include natural speech variations (multiple speakers, interviews, sound effects). The dataset was manually annotated with topic categories from the IPTC subject reference system. For this study, two categories were selected: "Politics" (a large category with more training data) and "Science" (a medium category with less training data).
- Phondat/Siemens100 Speech Database: Used for training the monophone acoustic models of the ASR system.
- German Parliament Corpus: Used for training the syllable-based language model for the ASR system.
Key Insights and Ideas
- Sub-word units are beneficial: Despite the potential for increased noise and loss of word-level discriminative power, using syllable-based features demonstrably improves classification performance, especially for categories with limited training data (like "Science").
- LSVMs excel with high dimensionality: The LSVM's ability to manage complexity independently of feature space size is crucial. It effectively leverages the expanded feature sets (higher-order n-grams) derived from sub-word units, whereas the Naive Bayes classifier tends to be overwhelmed by too many features.
- Robustness to ASR errors: The paper highlights that even with a high syllable error rate (around 70%) from a domain-independent ASR system, classifier performance doesn't degrade drastically. This suggests that classifiers can learn to compensate for ASR inaccuracies.
- Text-trained classifiers for audio: It's possible to train classifiers on text transcripts and achieve comparable performance on spoken documents, reducing the reliance on labeled audio data. This is particularly effective with LSVMs.
Key Results
- Text Classification: Using syllable 1-grams improved classification accuracy over word 1-grams for both "Politics" and "Science" topics. LSVMs generally outperformed Naive Bayes, especially with higher-order features. For "Science," syllable 1-grams with LSVM achieved an F-measure of 58.8%, significantly better than word 1-grams (19.7%).
- Spoken Document Classification: Even with a ~70% syllable error rate, the system achieved respectable performance. For "Politics," LSVM with syllable 2-grams reached an F-measure of 63.1%, while Naive Bayes achieved 56.0%. For "Science," LSVM with syllable 2-grams achieved 46.3%, outperforming Naive Bayes (26.0%).
- Text-trained on Audio: For the "Politics" topic, LSVMs trained on text transcripts achieved F-measures of 38.5% (syllable 1-grams) and 61.1% (syllable 2-grams) when classifying spoken documents. This is comparable to, or even better than, training directly on ASR output in some cases.
Limitations and Future Work
- Limitations:
- The Naive Bayes classifier showed sensitivity to data splits, indicating its limitations in controlling overfitting.
- While the system is robust to ASR errors, the syllable error rate remains high (around 70%), suggesting room for further improvement.
- Future Work:
- Explore further feature combinations and weighting/filtering schemes for LSVMs to optimize the feature set.
- Investigate applying feature weighting/filtering schemes to the Naive Bayes classifier.
- Further develop the syllable-based language model, potentially by optimizing the syllable vocabulary for better classification performance.
- Explore research directions that improve spoken document classification without necessarily reducing ASR error rates directly.
Summary updated