Automated bird sound recognition in realistic settings

Timos Papadopoulos, Stephen J. Roberts, Katherine J. WillisarXiv 2018 · arXiv

Abstract

While automated bird sound recognition has seen significant progress in laboratory settings, transitioning these systems to real-world applications remains a challenge. This paper presents a scalable framework for bird sound identification that moves away from manual data curation toward an automated pipeline using crowd-sourced audio. By implementing a probabilistic "rejection option" based on entropy, the system enhances reliability, ensuring that users receive fewer, but significantly more accurate, identifications.

Introduction

In the field of bioacoustics, identifying bird species by their vocalizations is a classic machine learning task. However, a significant gap exists between models that perform well on curated datasets of a few species and systems capable of functioning in the wild. Most research operates under ideal conditions: small species counts (5–20) and training data meticulously labeled at the syllable level by experts.

In contrast, a practical application must handle hundreds of species and process noisy, unsegmented audio. This research addresses these challenges by focusing on scalability, automated training, and the critical ability of a classifier to recognize its own uncertainty.

Methodology

The Automated Pipeline

To achieve global scalability, the authors developed a "blueprint" that replaces manual intervention with an automated pipeline. The system utilizes a "power threshold" method to automatically detect and extract bird sounds from long recordings, stripping away silence and background noise. This allows the model to be trained on large-scale, crowd-sourced data without the need for expert segmentation.

Classification Strategies

The study compared two primary classification approaches:

  1. k-Nearest Neighbour (kNN): An instance-based learner using histogram-based features.
  2. Support Vector Machine (SVM): A standard classifier utilizing time-summarization features.

The authors highlight that kNN, often considered a "lazy" learner, offers distinct advantages for bioacoustics. Because it stores instances rather than learning a fixed model, new species or additional training data can be integrated on the fly without requiring a full retraining of the system.

Uncertainty and Rejection

A core innovation of this framework is the introduction of a probabilistic measure of uncertainty. Instead of forcing a prediction for every audio clip, the system calculates the entropy of the classifier's output. High entropy indicates that the model is confused between multiple species. By implementing a "rejection option," the system can choose not to provide a result when uncertainty is high, prioritizing precision over raw coverage.

Experimental Setup

To ensure real-world validity, the researchers utilized two entirely separate datasets:

  • Training Data: Derived from xeno-canto (XC), a massive crowd-sourced collection. The authors used over 6,000 recordings covering 99 species, relying solely on community-provided labels.
  • Testing Data: Derived from the Animal Sound Archive (RAV). This hand-curated, high-quality dataset was used to simulate a user providing a clean audio excerpt to a mobile application.

Key Results

The system was tested on scales of 72 and 99 species—significantly larger than the datasets typical in existing literature. While "Top-1" accuracy remained modest (9–16%), the Accuracy@10 (the correct species appearing within the top 10 results) reached 40–50%.

Furthermore, the entropy-based rejection criterion proved highly effective. The Mean Reciprocal Rank (MRR)—which measures how high the correct result appears in the ranked list—improved consistently as the rejection threshold was tightened. At the highest rejection rates, where the system only returned results for the most certain samples, it achieved a perfect MRR of 1.0.

Discussion and Future Work

The findings suggest that simple features, such as one-dimensional histograms of mode frequency, can perform as well as complex features, which is beneficial for the limited computational power of mobile devices. However, challenges remain regarding the memory footprint of kNN as it scales toward the 10,000 global bird species.

Future iterations of this work aim to incorporate modern deep learning architectures, such as Convolutional Neural Networks (CNNs), and integrate metadata like GPS coordinates. By using a phone’s location to filter candidate species based on local phenology, the system's accuracy and relevance for the end-user can be further enhanced.

Summary updated

Abstract

We evaluated the effectiveness of an automated bird sound identification system in a situation that emulates a realistic, typical application. We trained classification algorithms on a crowd-sourced collection of bird audio recording data and restricted our training methods to be completely free of manual intervention. The approach is hence directly applicable to the analysis of multiple species collections, with labelling provided by crowd-sourced collection. We evaluated the performance of the bird sound recognition system on a realistic number of candidate classes, corresponding to real conditions. We investigated the use of two canonical classification methods, chosen due to their widespread use and ease of interpretation, namely a k Nearest Neighbour (kNN) classifier with histogram-based features and a Support Vector Machine (SVM) with time-summarisation features. We further investigated the use of a certainty measure, derived from the output probabilities of the classifiers, to enhance the interpretability and reliability of the class decisions. Our results demonstrate that both identification methods achieved similar performance, but we argue that the use of the kNN classifier offers somewhat more flexibility. Furthermore, we show that employing an outcome certainty measure provides a valuable and consistent indicator of the reliability of classification results. Our use of generic training data and our investigation of probabilistic classification methodologies that can flexibly address the variable number of candidate species/classes that are expected to be encountered in the field, directly contribute to the development of a practical bird sound identification system with potentially global application. Further, we show that certainty measures associated with identification outcomes can significantly contribute to the practical usability of the overall system.
BibTeX
@misc{papadopoulos2018automated,
  author = {Timos Papadopoulos and Stephen J. Roberts and Katherine J. Willis},
  title = {{Automated bird sound recognition in realistic settings}},
  year = {2018},
  eprint = {1809.01133},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1809.01133},
}
Figure