TL;DR
This paper introduces "Fused Visual Features" (FVFs) by combining two types of visual representations derived from acoustic event spectrograms: row vectors and SIFT features. These FVFs, processed using Locality-Constrained Linear Coding (LLC), demonstrate robust performance in classifying acoustic events, even in noisy environments, outperforming existing methods.
What Problem Does This Paper Address?
Acoustic Event Classification (AEC) is crucial for applications like surveillance and robotics, aiming to identify specific sounds within an environment. Traditional methods often rely on Mel-frequency cepstral coefficients (MFCCs), which are sensitive to background noise. While previous work has explored using visual features from spectrograms, the challenge remains to develop representations that are both discriminative and robust to noise. This paper tackles this by proposing a novel feature fusion technique that leverages the strengths of different visual feature extraction methods applied to a specialized representation of acoustic events.
Methods
The core of this paper lies in the proposed "Fused Visual Features" (FVFs). The process begins with generating a grayscale Gammatonegram from the acoustic event signal. This representation captures more spectral information than a standard spectrogram. Two types of visual features are then extracted from this Gammatonegram:
- Row Vectors: Intensity values from each row of the Gammatonegram are treated as feature vectors.
- SIFT Vectors: Scale-Invariant Feature Transform (SIFT) keypoints are detected on the Gammatonegram, and 128-dimensional feature vectors are extracted for each keypoint.
Both sets of features are then encoded using Locality-Constrained Linear Coding (LLC). LLC is an extension of sparse coding that assumes locality of features is more discriminative than pure sparsity. It maps each feature vector into a code based on a learned visual vocabulary, emphasizing nearby visual words.
Finally, the LLC-encoded features from the row vectors and SIFT vectors are fused (concatenated) to create the FVFs. This fusion aims to combine complementary information from both feature types.
Datasets
The proposed approach is evaluated on two publicly available datasets:
- UPC-TALP dataset: Contains 12 types of isolated acoustic events recorded in meeting room and office environments.
- DCASE-2013 dataset: Includes 16 types of isolated acoustic events.
To assess robustness, "speech babble" noise was added to the clean acoustic events at various Signal-to-Noise Ratios (SNRs) of 20dB, 10dB, and 0dB.
Key Insights and Ideas
- Gammatonegrams offer richer spectral information: The paper highlights that Gammatonegrams provide more detailed spectral information compared to standard spectrograms, making them a better basis for feature extraction.
- LLC's advantage over traditional sparse coding: By incorporating a locality constraint, LLC is shown to be more effective than pure sparse coding for acoustic event classification, as it better captures the discriminative nature of nearby features.
- Fusion is key to robustness: Neither row vectors nor SIFT vectors alone achieve optimal performance. Their combination into FVFs significantly boosts accuracy and robustness, suggesting that these two feature types capture complementary aspects of the acoustic events.
- SIFT's noise resilience: SIFT feature extraction, by blurring the Gammatonegram and localizing keypoints around strong peaks, effectively reduces the impact of noise and highlights discriminative event characteristics.
- Vocabulary size trade-off: The study explores the impact of vocabulary size on performance, finding that while larger vocabularies improve accuracy, they also increase computational complexity. A vocabulary size of 256 visual words was found to offer a good balance.
Key Results
The proposed Fused Visual Features (FVFs) combined with a linear SVM classifier achieved state-of-the-art performance:
- UPC-TALP dataset: An overall recognition accuracy of 96.40% in clean and noisy conditions.
- DCASE dataset: An overall recognition accuracy of 90.45% in clean and noisy conditions.
These results significantly outperform baseline methods like MFCCs, GTCCs, DNNs, CNNs, and the previously proposed Bag-of-Visual-Words (BoVW) approach, especially in noisy environments. For instance, on the UPC-TALP dataset at clean conditions, FVFs achieved 99.17% accuracy, surpassing the BoVW method's 93.54%.
Limitations and Future Work
- DCASE dataset quality: The authors note that the DCASE dataset's audio recordings are inherently noisy, which leads to lower performance compared to the UPC-TALP dataset, even in "clean" conditions.
- Future work: The authors suggest exploring the concatenation of spectral features with other image features to FVFs to further enhance performance. They also mention that while a vocabulary size of 256 offers a good balance, further investigation into optimal vocabulary selection could be beneficial.
Summary updated