No Free Lunch from Audio Pretraining in Bioacoustics: A Benchmark Study of Embeddings

Chenggang Chen, Zhiyu YangarXiv 2025 · DOI · arXiv

Introduction

In computational bioacoustics, "off-the-shelf" pretrained audio models are frequently employed as universal feature extractors. The prevailing assumption is that models trained on massive general audio datasets (e.g., YouTube, human speech) have learned a robust "language of audio" applicable to animal sound classification. However, bioacoustic environments are characterized by significant noise—wind, rain, and overlapping species—which can degrade performance. This study investigates the efficacy of these embeddings, revealing that without task-specific fine-tuning, pretrained models often fail to distinguish target signals from background noise.

Methodology

The researchers evaluated 11 deep learning architectures, including ResNet variants (scratch and ImageNet-pretrained), AlexNet, VGGish (audio-pretrained), and AVES (bioacoustics-pretrained). The study moved beyond simple accuracy, analyzing embedding quality through a multi-step pipeline:

  1. Feature Extraction: Generating high-dimensional embeddings for each audio sample.
  2. Dimensionality Reduction: Projecting features into 2D space using t-SNE and UMAP for visualization.
  3. Clustering: Applying K-Means to these projections.
  4. Quantitative Evaluation: Measuring cluster-label alignment using Normalized Mutual Information (NMI), Adjusted Random Index (ARI), and Silhouette coefficients.

Datasets

The Benchmark of Animal Sounds (BEANS) collection provided ten datasets:

  • Classification: Watkins marine mammals (wtkn), Egyptian fruit bats (bats), Cornell birds (cbi), domestic dogs (dogs), and mosquito wing beats (hbdb).
  • Detection: Mammal/bird calls (dcase), Eastern North American birds (enab), minke whales (hiceas), rainforest frogs/birds (rfcx), and Hainan gibbons (gib). Detection tasks are notably harder due to the prevalence of unlabeled background noise (Class 0).

Key Insights: The Impact of Fine-Tuning and Background Noise

A critical finding is the "background trap": audio-pretrained models often fail to separate animal signals from environmental noise in their latent space. Without fine-tuning, these models frequently cluster background noise with target sounds, leading to poor discriminative power.

t-SNE visualizations showing the effect of fine-tuning on embedding separation across different datasets and models.

Fine-tuning proved essential; NMI scores typically increased by over 1.5x after full training. Interestingly, ResNet models pretrained on ImageNet sometimes outperformed audio-pretrained models in detection tasks, suggesting that visual feature extraction may offer better structural robustness against acoustic noise than current audio-pretraining paradigms.

Results

Performance varied significantly between task types. While audio-pretrained models like AVES excelled in clean classification tasks (often >0.8 accuracy), they struggled in detection, frequently yielding precision below 0.4.

Comprehensive performance metrics for various models across the BEANS benchmark datasets.

Quantitative analysis showed that "cleaning" data by removing unlabeled background segments significantly boosted performance. For instance, the Average Precision for the aves-bio model on the gibbon dataset improved from 0.284 to 0.457 after such cleaning.

Conclusion and Future Work

The study concludes that there is "no free lunch" when using pretrained audio embeddings for bioacoustics. Success requires either extensive fine-tuning or rigorous data cleaning. Future research should prioritize developing architectures specifically designed for noisy acoustic environments and exploring more effective data augmentation strategies to mitigate class imbalance.

Summary updated

Abstract

Bioacoustics, the study of animal sounds, offers a non-invasive method to monitor ecosystems. Extracting embeddings from audio-pretrained deep learning (DL) models without fine-tuning has become popular for obtaining bioacoustic features for tasks. However, a recent benchmark study reveals that while fine-tuned audio-pretrained VGG and transformer models achieve state-of-the-art performance in some tasks, they fail in others. This study benchmarks 11 DL models on the same tasks by reducing their learned embeddings' dimensionality and evaluating them through clustering. We found that audio-pretrained DL models 1) without fine-tuning even underperform fine-tuned AlexNet, 2) both with and without fine-tuning fail to separate the background from labeled sounds, but ResNet does, and 3) outperform other models when fewer background sounds are included during fine-tuning. This study underscores the necessity of fine-tuning audio-pretrained models and checking the embeddings after fine-tuning. Our codes are available: https://github.com/NeuroscienceAI/Audio\_Embeddings
BibTeX
@misc{chen2025no,
  author = {Chenggang Chen and Zhiyu Yang},
  title = {{No Free Lunch from Audio Pretraining in Bioacoustics: A Benchmark Study of Embeddings}},
  year = {2025},
  eprint = {2508.10230},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.10230},
}
Figure