An empirical study of weakly supervised audio tagging embeddings for general audio representations

Heinrich Dinkel, Zhiyong Yan, Yongqing Wang, Junbo Zhang, Yujun Wang

Introduction

In the pursuit of universal artificial intelligence, the development of "foundation models"—backbones capable of generalizing across diverse tasks—remains a primary goal. While computer vision has long utilized ImageNet-pre-trained models as standard feature extractors, the audio domain remains fragmented. Research typically bifurcates into environmental sound classification and speech processing, with few models bridging the gap effectively.

This study investigates whether weakly supervised audio tagging (AT) can serve as a robust foundation for general audio representations. By training models on millions of audio clips with high-level tags (e.g., "dog bark," "speech") without precise temporal annotations, researchers from Xiaomi Corporation explored the potential of these embeddings to generalize across 14 distinct downstream tasks, ranging from music emotion to keyword spotting.

Methodology

The researchers focused on two lightweight, vision-inspired architectures: MobileNetV2 and EfficientNet-B0. These models were chosen for their efficiency, offering a significantly smaller footprint (3M to 4.6M parameters) compared to traditional heavyweights like CNN14 (80M parameters) while maintaining competitive performance.

Pre-training and Weak Supervision

The models were pre-trained on AudioSet, a massive repository containing approximately 5,200 hours of audio across 527 classes. The training utilized "weak labels," meaning the model was tasked with predicting the presence of a sound within a 10-second window without knowing exactly when the sound occurred. This approach forces the model to learn high-level semantic features rather than just local acoustic patterns.

Transfer Learning Strategies

To evaluate the versatility of the pre-trained embeddings, two primary transfer learning strategies were employed:

  1. Frozen Embeddings: The pre-trained model acts as a fixed feature extractor. Only a simple linear classifier is trained for the downstream task.
  2. Full Fine-tuning: The entire model’s weights are updated, allowing the backbone to adapt to the specific nuances of the new dataset.

Experimental Evaluation

The study’s breadth is a key strength, utilizing 14 downstream datasets categorized into two domains:

  • Sound Domain: Environmental noise (UrbanSound8k, ESC-50), general audio tagging (FSD series), music (NSynth), and emotion recognition (RAVDESS).
  • Speech Domain: Keyword spotting (Google Speech Commands), intent classification (Fluent Speech Commands), speaker identification (VoxCeleb1), and language identification (VoxLingua107).

The performance of these weakly supervised models was benchmarked against BYOL-A, a state-of-the-art self-supervised learning (SSL) method.

Results and Discussion

Dominance in the Sound Domain

The weakly supervised models demonstrated clear superiority in sound-related tasks. On the FSD50k dataset, the EfficientNet-B0 reached a mean Average Precision (mAP) of 57.20, significantly outperforming BYOL-A’s 40.38. This suggests that the semantic signal provided by AudioSet labels allows the model to develop a more sophisticated understanding of sound categories than SSL methods, which often focus on more general acoustic patterns.

The Speech-Sound Gap

A critical finding of the study is the performance discrepancy in speech tasks. When used as "frozen" feature extractors, the models performed poorly on speech-related benchmarks. The authors attribute this to the "Speech" label in AudioSet; because the model only learns to identify that speech is occurring (to satisfy the broad label), it fails to capture the granular phonetic and speaker-specific details necessary for tasks like keyword spotting or speaker identification. This creates a "semantic bucket" where all speech is categorized similarly regardless of content.

However, fine-tuning proved to be a powerful remedy. When the entire model was updated, the performance on the Fluent Speech Commands (FSC) task surged to approximately 98% accuracy, matching or exceeding specialized baselines. This indicates that the pre-trained weights provide a useful starting point, even if the frozen embeddings lack granularity.

Efficiency and Scalability

Despite their smaller size, the MobileNetV2 and EfficientNet-B0 architectures achieved results comparable to much larger models. This efficiency makes them ideal candidates for on-device applications. However, the study also noted that on extremely large speech datasets (e.g., VoxLingua107), training from scratch still outperformed transfer learning, indicating that the benefits of AudioSet pre-training may diminish as the target dataset size increases.

Conclusion

This empirical study confirms that weakly supervised audio tagging provides a potent foundation for general audio representations, particularly for environmental and musical tasks. While the "semantic gap" prevents frozen embeddings from excelling in speech tasks, full parameter fine-tuning allows these models to become highly effective speech processors. Future research may focus on incorporating more granular speech data into the pre-training phase to create a truly universal audio backbone that requires minimal adaptation.

Summary updated

BibTeX
@inproceedings{dinkel2022empirical,
  author = {Heinrich Dinkel and Zhiyong Yan and Yongqing Wang and Junbo Zhang and Yujun Wang},
  title = {{An empirical study of weakly supervised audio tagging embeddings for general audio representations}},
  booktitle = {Odyssey},
  year = {2022},
  doi = {10.21437/odyssey.2022-54},
}
Figure