Abstract
Convolutional Neural Networks (CNNs) have proven very effective in image classification and show promise for audio. We use various CNN architectures to classify the soundtracks of a dataset of 70M training videos (5.24 million hours) with 30,871 video-level labels. We examine fully connected Deep Neural Networks (DNNs), AlexNet, VGG, Inception, and ResNet. We investigate varying the size of both training set and label vocabulary, finding that analogs of the CNNs used in image classification do well on our audio classification task, and larger training and label sets help up to a point. A model using embeddings from these classifiers does much better than raw features on the Audio Set Acoustic Event Detection (AED) classification task.
Introduction
While deep Convolutional Neural Networks (CNNs) have revolutionized computer vision, audio classification—specifically Acoustic Event Detection (AED)—has historically lacked similar large-scale breakthroughs. This research explores whether image-centric CNN architectures can be effectively adapted for audio, how performance scales with massive datasets and broad label vocabularies, and the utility of learned embeddings for downstream tasks.
Methodology and Datasets
The researchers treated audio as a 2D image by converting signals into log-mel spectrograms. These were divided into 96x64 "patches" (representing 960ms of audio) and fed into several standard architectures:
- Fully Connected DNN: Baseline model.
- AlexNet, VGG, Inception-V3, and ResNet-50: Standard image classification models.
Training utilized two primary datasets:
- YouTube-100M: 70 million videos (5.24M hours) with 30,871 machine-generated "weak" labels.
- Audio Set: Over 1 million 10-second excerpts used for human-labeled benchmarking.
Key Insights
- Spectrograms as Images: Standard CNNs are remarkably effective at extracting features from spectrograms, despite time and frequency axes representing different physical properties.
- The Power of Embeddings: Internal representations learned by ResNet-50 on YouTube-100M significantly boosted performance on Audio Set, more than doubling accuracy compared to raw features.
- Label Breadth as Regularization: Training on a wider variety of labels (30K vs 3K) improved generalization, suggesting that learning diverse concepts helps the model capture more robust audio features.
- Weak Labels Work: Even though labels were noisy and lacked precise timestamps, the models successfully identified relevant acoustic cues given sufficient data.
Key Results
Modern, deep architectures significantly outperformed simpler models. ResNet-50 achieved the highest performance with an AUC of 0.926 and a mean Average Precision (mAP) of 0.212 on the 3K label set. Performance scaled steadily with data size, plateauing around 7 million videos.
Limitations and Future Work
Current models use simple averaging for temporal aggregation; the authors suggest that RNNs or LSTMs could better model temporal dependencies. Additionally, addressing machine-generated label noise and the high computational cost of training remain open challenges.
Summary updated