Introduction
In computational bioacoustics, "off-the-shelf" pretrained audio models are frequently employed as universal feature extractors. The prevailing assumption is that models trained on massive general audio datasets (e.g., YouTube, human speech) have learned a robust "language of audio" applicable to animal sound classification. However, bioacoustic environments are characterized by significant noise—wind, rain, and overlapping species—which can degrade performance. This study investigates the efficacy of these embeddings, revealing that without task-specific fine-tuning, pretrained models often fail to distinguish target signals from background noise.
Methodology
The researchers evaluated 11 deep learning architectures, including ResNet variants (scratch and ImageNet-pretrained), AlexNet, VGGish (audio-pretrained), and AVES (bioacoustics-pretrained). The study moved beyond simple accuracy, analyzing embedding quality through a multi-step pipeline:
- Feature Extraction: Generating high-dimensional embeddings for each audio sample.
- Dimensionality Reduction: Projecting features into 2D space using t-SNE and UMAP for visualization.
- Clustering: Applying K-Means to these projections.
- Quantitative Evaluation: Measuring cluster-label alignment using Normalized Mutual Information (NMI), Adjusted Random Index (ARI), and Silhouette coefficients.
Datasets
The Benchmark of Animal Sounds (BEANS) collection provided ten datasets:
- Classification: Watkins marine mammals (
wtkn), Egyptian fruit bats (bats), Cornell birds (cbi), domestic dogs (dogs), and mosquito wing beats (hbdb). - Detection: Mammal/bird calls (
dcase), Eastern North American birds (enab), minke whales (hiceas), rainforest frogs/birds (rfcx), and Hainan gibbons (gib). Detection tasks are notably harder due to the prevalence of unlabeled background noise (Class 0).
Key Insights: The Impact of Fine-Tuning and Background Noise
A critical finding is the "background trap": audio-pretrained models often fail to separate animal signals from environmental noise in their latent space. Without fine-tuning, these models frequently cluster background noise with target sounds, leading to poor discriminative power.
Fine-tuning proved essential; NMI scores typically increased by over 1.5x after full training. Interestingly, ResNet models pretrained on ImageNet sometimes outperformed audio-pretrained models in detection tasks, suggesting that visual feature extraction may offer better structural robustness against acoustic noise than current audio-pretraining paradigms.
Results
Performance varied significantly between task types. While audio-pretrained models like AVES excelled in clean classification tasks (often >0.8 accuracy), they struggled in detection, frequently yielding precision below 0.4.
Quantitative analysis showed that "cleaning" data by removing unlabeled background segments significantly boosted performance. For instance, the Average Precision for the aves-bio model on the gibbon dataset improved from 0.284 to 0.457 after such cleaning.
Conclusion and Future Work
The study concludes that there is "no free lunch" when using pretrained audio embeddings for bioacoustics. Success requires either extensive fine-tuning or rigorous data cleaning. Future research should prioritize developing architectures specifically designed for noisy acoustic environments and exploring more effective data augmentation strategies to mitigate class imbalance.
Summary updated