Semantically-Informed Deep Neural Networks For Sound Recognition

Michele Esposito, Giancarlo Valente, Yenisel Plasencia Calaña, Michel Dumontier, Bruno L. Giordano, Elia Formisano

Abstract

Traditional sound recognition systems treat categorical labels as independent entities, failing to capture the inherent semantic relationships between sounds. We present semDNN, a deep neural network architecture that maps audio inputs to continuous semantic embeddings (word2vec) rather than discrete one-hot labels. Our results demonstrate that semDNN not only improves classification accuracy across multiple benchmarks but also develops internal representations that align significantly better with human auditory perception than standard categorical models.

Introduction

In standard sound classification, the labels "dog barking" and "cat meowing" are treated as mathematically equidistant from "chainsaw." This reliance on one-hot encoding ignores the semantic proximity of biological sounds compared to mechanical ones. Human listeners, by contrast, leverage high-level semantic categories to group and interpret sounds. This paper addresses the gap between categorical AI models and human-like "semantic intelligence" in sound recognition.

Methodology

The researchers compared two architectures: catDNN (Categorical) and semDNN (Semantic). Both are based on the VGGish architecture, processing log-mel spectrograms through four convolutional blocks.

Comparison of label encoding strategies and the catDNN/semDNN architecture. (a) shows the transformation of labels into one-hot vs. word2vec vectors. (b) and (c) visualize the resulting embedding spaces. The right panel details the shared convolutional architecture.

The critical distinction lies in the output layer:

  • catDNN: Uses a 9,960-unit layer with sigmoid activation for standard multi-label classification.
  • semDNN: Uses a 300-unit layer with linear activation to predict a word2vec embedding representing the linguistic meaning of the sound.

Models were trained on a semantically balanced subset of the Super Hard Drive Combo (approx. 388k sounds) and evaluated using Representational Similarity Analysis (RSA) against human behavioral data.

Key Results and Discussion

Performance and Generalization

semDNN consistently outperformed catDNN across five datasets, including external benchmarks like FSD50k and ESC-50. By learning the "meaning space," semDNN exhibits a "semantic safety net"—errors are more likely to be semantically related (e.g., confusing a violin with a cello) rather than acoustically similar but semantically distant.

Alignment with Human Perception

A primary finding is the superior alignment of semDNN with human auditory grouping. In RSA tests, semDNN achieved an $R^2_{CV}$ of 0.371, nearly doubling the performance of catDNN (0.199) and surpassing industry standards like Google’s Yamnet (0.302). Visualizations using t-SNE confirm that semDNN naturally organizes sounds into macro-classes (nature, human, music) without explicit supervision.

Conclusion

Transitioning from categorical to semantic label encoding allows deep neural networks to bridge the gap between raw acoustic features and conceptual meaning. Future work will explore more advanced embeddings like BERT and the integration of formal sound ontologies to handle complex, overlapping auditory scenes.

Summary updated

Abstract

Deep neural networks (DNNs) for sound recognition learn to categorize a barking sound as a "dog" and a meowing sound as a "cat" but do not exploit information inherent to the semantic relations between classes (e.g., both are animal vocalisations). Cognitive neuroscience research, however, suggests that human listeners automatically exploit higher-level semantic information on the sources besides acoustic information. Inspired by this notion, we introduce here a DNN that learns to recognize sounds and simultaneously learns the semantic relation between the sources (semDNN). Comparison of semDNN with a homologous network trained with categorical labels (catDNN) revealed that semDNN produces semantically more accurate labelling than catDNN in sound recognition tasks and that semDNN-embeddings preserve higherlevel semantic relations between sound sources. Importantly, through a model-based analysis of human dissimilarity ratings of natural sounds, we show that semDNN approximates the behaviour of human listeners better than catDNN and several other DNN and NLP comparison models.
BibTeX
@inproceedings{esposito2023semantically,
  author = {Michele Esposito and Giancarlo Valente and Yenisel Plasencia Calaña and Michel Dumontier and Bruno L. Giordano and Elia Formisano},
  title = {{Semantically-Informed Deep Neural Networks For Sound Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095606},
}
Figure