SSL-Net: A Synergistic Spectral and Learning-Based Network for Efficient Bird Sound Classification

Yiyuan Yang, Kaichen Zhou, Niki Trigoni, Andrew Markham

TL;DR

SSL-Net is a novel framework that combines traditional acoustic signal processing (spectral features) with deep learning (pretrained models) to classify bird sounds with high accuracy and minimal data. By using a "synergistic" two-branch approach and clever feature fusion, it outperforms standard models while remaining efficient enough for real-world ecological monitoring.

What Problem Does This Paper Address?

Automated bird sound classification is vital for biodiversity monitoring, but it faces two major hurdles:

  1. The "Data Hunger" of Deep Learning: Training high-performing models from scratch requires massive amounts of labeled audio, which is expensive and time-consuming to produce.
  2. Domain Bias: While general-purpose pretrained audio models (like those trained on human speech or urban noise) exist, they often struggle with the specific nuances of avian vocalizations, which can be incredibly diverse within a single species or confusingly similar across different ones.

The authors sought to create a "label-efficient" model that leverages the power of big-data pretraining without losing the fine-grained acoustic details necessary for identifying specific birds.

Methods

The core of the paper is the SSL-Net architecture, which consists of two parallel branches:

  • The Learned Branch: This branch uses BEATs (an audio-pretrained transformer) as a backbone to extract high-level semantic features from raw waveforms. These features are then processed through a ResNet18 encoder.
  • The Spectral Branch: To ground the model in acoustic physics, this branch extracts three classic "handcrafted" features: MelSpectrogram (MEL), Short-Time Fourier Transform (STFT), and Mel-Frequency Cepstral Coefficients (MFCC). These act as a "domain anchor" to correct the bias of the general-purpose learned branch.

The researchers then tested three ways to combine these branches:

  1. Fixed Fusion: Simple concatenation of features.
  2. Shared Fusion: An attention-like mechanism that uses a Multilayer Perceptron (MLP) to dynamically reweight features based on their importance.
  3. Sampling Fusion: A sophisticated stochastic approach using the Gumbel-Softmax distribution, which allows the model to "sample" from the features during training, leading to more robust representations.

Datasets

The model was evaluated using the Western Mediterranean Wetland Birds (WMWB) dataset.

  • Source: Derived from Xeno-canto (a popular crowdsourced bird sound database).
  • Scale: Includes 20 different bird species and over 200 minutes of audio.
  • Challenge: The authors specifically tested the model's performance with limited data, down to just 200 labeled samples per species.

Key Insights and Ideas

  • Spectral Features as a "Correction": The authors found that while pretrained models are great at general audio, they are "biased." Adding traditional spectral features (like MFCCs) provides the specific acoustic "fingerprint" of bird calls that general models might overlook.
  • Fusion Matters: You can't just smash two feature sets together and expect magic. The Shared Fusion strategy emerged as the "sweet spot," offering a significant boost in accuracy while keeping the model complexity manageable.
  • Label Efficiency: The framework demonstrates that you don't need millions of samples to get state-of-the-art results. By "synergizing" learned and spectral data, the model reaches high performance much faster than models trained from scratch.

Key Results

  • Accuracy Boost: SSL-Net (using Spectral + LEAF/BEATs) achieved an accuracy of 87.29% on the WMWB dataset, significantly higher than the ResNet50 baseline (~77.7%).
  • Robustness: Even when the number of labeled samples per category was halved, the fusion-based approaches maintained much higher F1-scores compared to using learned or spectral features in isolation.
  • Discriminative Power: t-SNE visualizations (a way to map high-dimensional data into 2D) showed that the fused features created much tighter, more distinct clusters for different bird species compared to the features from a single branch.

Limitations and Future Work

  • Computational Overhead: While the model is "efficient" in terms of labels, the dual-branch architecture and fusion modules (especially Sampling Fusion) increase the total parameter count and memory usage compared to a simple single-branch model.
  • Backbone Dependency: The performance is heavily dependent on the quality of the pretrained audio model (BEATs/LEAF). As better general audio models are released, SSL-Net would need to be updated to stay at the cutting edge.
  • Real-world Noise: While the dataset is field-collected, the paper doesn't deeply explore how the model handles extreme "out-of-distribution" noise (like heavy machinery or wind) that wasn't present in the training set. Future work could involve testing the model's "ruggedness" in even harsher environments.

Summary updated

Abstract

Efficient and accurate bird sound classification is of important for ecology, habitat protection and scientific research, as it plays a central role in monitoring the distribution and abundance of species. However, prevailing methods typically demand extensively labeled audio datasets and have highly customized frameworks, imposing substantial computational and annotation loads. In this study, we present an efficient and general framework called SSL-Net, which combines spectral and learned features to identify different bird sounds. Encouraging empirical results gleaned from a standard field-collected bird audio dataset validate the efficacy of our method in extracting features efficiently and achieving heightened performance in bird sound classification, even when working with limited sample sizes. Furthermore, we present three feature fusion strategies, aiding engineers and researchers in their selection through quantitative analysis.
BibTeX
@inproceedings{yang2024ssl,
  author = {Yiyuan Yang and Kaichen Zhou and Niki Trigoni and Andrew Markham},
  title = {{SSL-Net: A Synergistic Spectral and Learning-Based Network for Efficient Bird Sound Classification}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=k4BD6vn3ji},
}

Loading related content...

Figure