Unsupervised Learning of Semantic Audio Representations

Aren Jansen, Manoj Plakal, Ratheet Pandya, Daniel P. W. Ellis, Shawn Hershey, Jiayang Liu, R. Channing Moore, Rif A. Saurous

Abstract

Even in the absence of any explicit semantic annotation, vast collections of audio recordings provide valuable information for learning the categorical structure of sounds. We consider several class-agnostic semantic constraints that apply to unlabeled nonspeech audio: (i) noise and translations in time do not change the underlying sound category, (ii) a mixture of two sound events inherits the categories of the constituents, and (iii) the categories of events in close temporal proximity are likely to be the same or related. Without labels to ground them, these constraints are incompatible with classification loss functions. However, they may still be leveraged to identify geometric inequalities needed for triplet loss-based training of convolutional neural networks. The result is low-dimensional embeddings of the input spectrograms that recover 41% and 84% of the performance of their fully-supervised counterparts when applied to downstream query-by-example sound retrieval and sound event classification tasks, respectively. Moreover, in limited-supervision settings, our unsupervised embeddings double the state-of-the-art classification performance.

Introduction

Deep learning has revolutionized audio processing, but these models typically require massive amounts of manually labeled data. While raw audio is abundant, labeling it is expensive and time-consuming. This research addresses the challenge of learning a semantic representation of sound—where similar sound categories are mathematically close in a vector space—without explicit labels. By exploiting inherent structural properties of audio, such as the likelihood that sounds occurring close together in time are related, we can derive a training signal from unlabeled data.

Methods

The proposed method utilizes Triplet Loss-based Metric Learning to train a ResNet-50 architecture. The model maps audio spectrograms to 128-dimensional embeddings. Training involves "triplets" consisting of an Anchor (reference clip), a Positive (semantically similar clip), and a Negative (likely different clip). The loss function optimizes the network to minimize the distance between anchor-positive pairs while maximizing the distance to negative samples.

Positive examples are generated using four "class-agnostic" semantic constraints:

  1. Gaussian Noise: Adding noise to the anchor.
  2. Time/Frequency Translation: Shifting the anchor in time or pitch.
  3. Example Mixing: Mixing the anchor with another random sound.
  4. Temporal Proximity: Selecting a clip from the same recording within 10 seconds of the anchor.

Results and Discussion

The model was trained on roughly 10 million triplets from the unlabeled portion of AudioSet and evaluated on Query-by-Example (QbE) retrieval and Sound Event Classification.

Performance of Unsupervised Constraints

Temporal Proximity proved to be the most powerful individual constraint, recovering 74% of the supervised performance gap in classification. However, combining all four methods (Joint Training) yielded the best results, suggesting that different constraints provide complementary semantic information.

Table 1: Performance comparison of different unsupervised representations across retrieval and classification tasks.

As shown in Table 1, the Joint Unsupervised Triplet model achieved a mean Average Precision (mAP) of 0.575 in retrieval (recovering 41% of the gap) and 0.259 in classification (recovering 84% of the supervised topline).

Data Efficiency in Limited-Supervision Settings

A key advantage of unsupervised pre-training is its performance in scenarios with scarce labeled data. In "lightly-supervised" settings with only 20 labels per class (0.5% of the dataset), the unsupervised embeddings doubled the performance of standard models trained from scratch.

Table 2: Comparison of mAP in limited-supervision settings (0.5% of labels).

Conclusion

This work demonstrates that semantic audio representations can be effectively learned from unlabeled data by leveraging geometric constraints. While a performance gap remains in retrieval tasks, the high recovery rate in classification and the significant gains in low-data regimes highlight the practical utility of unsupervised triplet loss training for large-scale audio analysis. Future work may explore multi-modal signals, such as combining audio with visual data from the same video sources.

Summary updated

Abstract

Even in the absence of any explicit semantic annotation, vast collections of audio recordings provide valuable information for learning the categorical structure of sounds. We consider several class-agnostic semantic constraints that apply to unlabeled nonspeech audio: (i) noise and translations in time do not change the underlying sound category, (ii) a mixture of two sound events inherits the categories of the constituents, and (iii) the categories of events in close temporal proximity are likely to be the same or related. Without labels to ground them, these constraints are incompatible with classification loss functions. However, they may still be leveraged to identify geometric inequalities needed for triplet loss-based training of convolutional neural networks. The result is low-dimensional embeddings of the input spectrograms that recover 41% and 84% of the performance of their fully-supervised counterparts when applied to downstream query-by-example sound retrieval and sound event classification tasks, respectively. Moreover, in limited-supervision settings, our unsupervised embeddings double the state-of-the-art classification performance.
BibTeX
@inproceedings{jansen2018unsupervised,
  author = {Aren Jansen and Manoj Plakal and Ratheet Pandya and Daniel P. W. Ellis and Shawn Hershey and Jiayang Liu and R. Channing Moore and Rif A. Saurous},
  title = {{Unsupervised Learning of Semantic Audio Representations}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=BlgC7K9tVM},
}
Figure