Abstract
Traditional sound recognition systems treat categorical labels as independent entities, failing to capture the inherent semantic relationships between sounds. We present semDNN, a deep neural network architecture that maps audio inputs to continuous semantic embeddings (word2vec) rather than discrete one-hot labels. Our results demonstrate that semDNN not only improves classification accuracy across multiple benchmarks but also develops internal representations that align significantly better with human auditory perception than standard categorical models.
Introduction
In standard sound classification, the labels "dog barking" and "cat meowing" are treated as mathematically equidistant from "chainsaw." This reliance on one-hot encoding ignores the semantic proximity of biological sounds compared to mechanical ones. Human listeners, by contrast, leverage high-level semantic categories to group and interpret sounds. This paper addresses the gap between categorical AI models and human-like "semantic intelligence" in sound recognition.
Methodology
The researchers compared two architectures: catDNN (Categorical) and semDNN (Semantic). Both are based on the VGGish architecture, processing log-mel spectrograms through four convolutional blocks.
The critical distinction lies in the output layer:
- catDNN: Uses a 9,960-unit layer with sigmoid activation for standard multi-label classification.
- semDNN: Uses a 300-unit layer with linear activation to predict a word2vec embedding representing the linguistic meaning of the sound.
Models were trained on a semantically balanced subset of the Super Hard Drive Combo (approx. 388k sounds) and evaluated using Representational Similarity Analysis (RSA) against human behavioral data.
Key Results and Discussion
Performance and Generalization
semDNN consistently outperformed catDNN across five datasets, including external benchmarks like FSD50k and ESC-50. By learning the "meaning space," semDNN exhibits a "semantic safety net"—errors are more likely to be semantically related (e.g., confusing a violin with a cello) rather than acoustically similar but semantically distant.
Alignment with Human Perception
A primary finding is the superior alignment of semDNN with human auditory grouping. In RSA tests, semDNN achieved an $R^2_{CV}$ of 0.371, nearly doubling the performance of catDNN (0.199) and surpassing industry standards like Google’s Yamnet (0.302). Visualizations using t-SNE confirm that semDNN naturally organizes sounds into macro-classes (nature, human, music) without explicit supervision.
Conclusion
Transitioning from categorical to semantic label encoding allows deep neural networks to bridge the gap between raw acoustic features and conceptual meaning. Future work will explore more advanced embeddings like BERT and the integration of formal sound ontologies to handle complex, overlapping auditory scenes.
Summary updated