TL;DR
This paper introduces a robust two-stage audio classification and segmentation method for video analysis. It effectively distinguishes between speech, music, environment sound, and silence using novel features like band periodicity and noise frame ratio, achieving high accuracy.
What Problem Does This Paper Address?
The ever-increasing volume of audio data, particularly within video content, necessitates efficient methods for automatically segmenting and classifying audio streams. This is crucial for applications like audio retrieval and, more specifically, for understanding video structure. While previous research has made strides in distinguishing between speech and music, accurately classifying a broader range of audio classes, including environment sounds and silence, especially within short time windows, remains a challenge. Existing methods often struggle with smaller window sizes or the complexity of diverse environment sounds.
Methods
The proposed method employs a two-stage classification approach:
- Speech/Non-Speech Discrimination:
- Pre-classification: A fast K-Nearest Neighbor (KNN) classifier is used, leveraging features like High Zero-Crossing Rate Ratio (HZCRR), Low Short-Time Energy Ratio (LSTER), and Spectrum Flux (SF). These features are computationally inexpensive and capture basic signal variations.
- Refinement: To improve accuracy, especially for noisy speech, Linear Spectral Pairs (LSP) distance analysis is employed. LSP features are known to be robust in noisy environments. The method compares the LSP covariance matrix of a test audio clip against pre-trained speech LSP covariance models.
- Non-Speech Classification:
- Silence Detection: A simple yet effective method based on average short-time energy and zero-crossing rate is used to identify silence segments.
- Music vs. Environment Sound Discrimination: For non-silence segments, a rule-based scheme is applied using Band Periodicity (BP), Spectrum Flux (SF), and Noise Frame Ratio (NFR). Band periodicity is a key feature, with music generally exhibiting higher periodicity than environment sounds. SF and NFR are used to handle overlaps and specific cases, such as tone-like environment sounds.
The paper also introduces novel features:
- Band Periodicity (BP): Measures the periodicity of specific sub-bands (500-1000Hz, 1000-2000Hz, etc.) using correlation analysis.
- Noise Frame Ratio (NFR): The proportion of frames identified as noise within an audio clip.
The audio stream is processed in one-second windows, further divided into 25ms non-overlapping frames for feature extraction. A post-processing smoothing step is also applied to ensure continuity and prevent rapid, unrealistic class changes.
Datasets
The paper utilizes a custom database for training and evaluation, compiled from:
- MPEG-7 test set CD1
- News and movie clips
- Internet audio clips
- Audio clips from TV studio, telephone (4kHz bandwidth), and 8kHz bandwidth speeches.
- Music content primarily consists of songs (pop music).
- Environment sound clips include a variety of sounds like automobile, beep, air plane, city life, combat, office, and house.
The training data comprises approximately 2 hours of audio, while the testing data includes 9587 seconds of speech, 3417 seconds of music, and 1201 seconds of environment sound. The speech clips have a ratio of pure speech to noisy speech of approximately 9:1.
Key Insights and Ideas
- Two-stage approach is effective: Separating speech/non-speech first and then classifying non-speech segments significantly improves overall accuracy and modularity.
- LSP for noisy speech refinement: LSP distance analysis proves crucial for accurately classifying noisy speech, which is often misclassified by simpler features.
- Band Periodicity as a strong music/environment sound differentiator: This novel feature, along with SF and NFR, provides a robust way to distinguish between harmonic music and more random environment sounds.
- Importance of feature combinations: No single feature is perfect; combining HZCRR, LSTER, SF, LSP, BP, and NFR in a structured manner leads to superior classification performance.
- Rule-based classification for non-speech: A well-defined set of rules, informed by experimental observations of feature distributions, effectively categorizes non-speech audio.
Key Results
The proposed method achieves high classification accuracy:
- Overall accuracy for speech, music, and environment sound: 96.51%.
- Speech/music discrimination accuracy: 98.03%.
- Refined noisy speech discrimination: Accuracy improved from 73.62% to 85.18% after LSP refinement.
- Environment sound classification: Accuracy improved to 79.27% after the full scheme.
- Time complexity: The segmentation and classification process takes approximately 15% of the audio clip's duration on a Pentium III 667MHz PC, making it suitable for real-time processing.
Limitations and Future Work
- Environment sound discrimination difficulty: The paper acknowledges that distinguishing environment sounds from music and speech remains challenging. An experiment testing whole clips (2-8 seconds) for environment sound discrimination showed an accuracy of 91.47%, with some misclassifications of crowd sounds as music and animal sounds as speech due to similar periodicity or vocal track shapes.
- Future work: The authors plan to:
- Improve the classification scheme to discriminate more audio classes.
- Focus on developing more effective schemes to apply audio content analysis for enhanced video structure parsing and indexing.
Summary updated