Unsupervised Learning of Semantic Audio Representations

Aren Jansen, Manoj Plakal, Ratheet Pandya, Daniel P. W. Ellis, Shawn Hershey, Jiayang Liu, R. Channing Moore, Rif A. Saurous

TL;DR

This paper demonstrates how to train powerful audio embedding models (ResNet-50) using millions of unlabeled clips by leveraging "class-agnostic" semantic constraints. By using triplet loss to enforce that related sounds (like those occurring close together in time) stay near each other in a vector space, the authors recover up to 84% of the performance of fully supervised models on classification tasks.

What Problem Does This Paper Address?

Deep learning has revolutionized audio processing, but these "data-hungry" models typically require massive amounts of manually labeled data. While collecting raw audio is easy (think YouTube or ambient sensors), labeling it is notoriously expensive and time-consuming.

The challenge is: How can we learn a "semantic" representation of sound—one where a dog bark is mathematically similar to another dog bark—without ever being told what a "dog" is?

The authors argue that sound has inherent structural properties (constraints) that we can exploit. For example, if you hear a sound and then hear another sound two seconds later in the same recording, they are likely related to the same environment or event. This paper explores how to turn these "common sense" observations into a formal training signal.

Methods

The researchers use Triplet Loss-based Metric Learning. The goal is to learn a mapping function (a CNN) that takes an audio spectrogram and outputs a 128-dimensional vector (embedding).

The training works by feeding the network "triplets":

  1. Anchor ($x_a$): A reference audio clip.
  2. Positive ($x_p$): A clip that should be semantically similar to the anchor.
  3. Negative ($x_n$): A clip that is likely different.

The loss function forces the network to move the anchor and positive closer together in the embedding space while pushing the negative further away. The "secret sauce" is how they select the Positive examples without labels:

  • Gaussian Noise: The positive is just the anchor with added noise.
  • Time/Frequency Translation: The positive is the anchor shifted slightly in time or pitch.
  • Example Mixing: The positive is created by mixing the anchor with another random sound.
  • Temporal Proximity: The positive is a clip taken from the same recording within 10 seconds of the anchor.

They use a ResNet-50 architecture, a heavy-duty model usually reserved for supervised image or audio tasks, to see if these unsupervised signals are strong enough to "fill" such a complex model.

Datasets

The primary dataset is AudioSet, a massive collection of over 2 million 10-second YouTube clips.

  • Training: They sampled roughly 10 million triplets from the unlabeled portions of the dataset.
  • Evaluation: They used the manually labeled portion of AudioSet to test two downstream tasks: Query-by-Example (QbE) retrieval (finding similar sounds) and Sound Event Classification (predicting one of 527 categories).

Key Insights and Ideas

  • Temporal Proximity is King: Of all the unsupervised constraints, "Temporal Proximity" (assuming sounds near each other are related) was the most powerful. It alone recovered 74% of the supervised performance gap in classification.
  • Constraints are Complementary: While temporal proximity is strong, combining all four methods (Joint Training) yielded the best results. This suggests that different constraints teach the network different things—e.g., noise invariance vs. environmental context.
  • Unsupervised Pre-training Boosts Small Data: In "lightly-supervised" scenarios (only 20 labels per class), the unsupervised embeddings doubled the performance of models trained from scratch. This is a huge win for niche audio applications where labels are scarce.
  • Metric Learning > Autoencoders: Unlike traditional unsupervised methods like autoencoders, which try to reconstruct the input, triplet loss focuses on relationships. This allows the model to ignore low-level acoustic details and focus on high-level semantic meaning.

Key Results

The authors compared their "Joint Unsupervised Triplet" model against a "Log Mel Spectrogram" baseline and a "Fully Supervised" topline:

  • Retrieval (mAP): The unsupervised model achieved a mean Average Precision (mAP) of 0.575, recovering 41% of the gap between the raw baseline (0.423) and the supervised topline (0.790).
  • Classification (mAP): Using a simple 2-layer classifier on top of the unsupervised embeddings reached 0.259 mAP. This recovered 84% of the performance of a fully supervised ResNet-50 (0.289).
  • Data Efficiency: With only 0.5% of the available labels, the unsupervised representation achieved 0.143 mAP, significantly outperforming a ResNet-50 trained from scratch on that same small subset (0.072).

Limitations and Future Work

  • The "Proximity" Assumption: Temporal proximity assumes a relatively stable acoustic environment. In very "busy" recordings with rapid-fire unrelated sounds, this constraint might introduce noise into the training.
  • Retrieval Gap: While classification performance was nearly matched, the gap in retrieval (41% recovery) remains significant. This suggests that while the model learns categories well, it may not yet capture the fine-grained nuances needed for perfect "query-by-example" matching.
  • Future Directions: The authors suggest exploring source-dependent weights for different constraints and investigating how these audio embeddings might be combined with visual signals (from the same YouTube videos) for even stronger multi-modal unsupervised learning.

Summary updated

Abstract

Even in the absence of any explicit semantic annotation, vast collections of audio recordings provide valuable information for learning the categorical structure of sounds. We consider several class-agnostic semantic constraints that apply to unlabeled nonspeech audio: (i) noise and translations in time do not change the underlying sound category, (ii) a mixture of two sound events inherits the categories of the constituents, and (iii) the categories of events in close temporal proximity are likely to be the same or related. Without labels to ground them, these constraints are incompatible with classification loss functions. However, they may still be leveraged to identify geometric inequalities needed for triplet loss-based training of convolutional neural networks. The result is low-dimensional embeddings of the input spectrograms that recover 41% and 84% of the performance of their fully-supervised counterparts when applied to downstream query-by-example sound retrieval and sound event classification tasks, respectively. Moreover, in limited-supervision settings, our unsupervised embeddings double the state-of-the-art classification performance.
BibTeX
@inproceedings{jansen2018unsupervised,
  author = {Aren Jansen and Manoj Plakal and Ratheet Pandya and Daniel P. W. Ellis and Shawn Hershey and Jiayang Liu and R. Channing Moore and Rif A. Saurous},
  title = {{Unsupervised Learning of Semantic Audio Representations}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=BlgC7K9tVM},
}

Loading related content...

Figure