Whombat: An open‐source audio annotation tool for machine learning assisted bioacoustics

Santiago Martínez Balvanera, Oisin Mac Aodha, Matthew J. Weldy, Holly Pringle, Ella Browning, Kate E. JonesMethods 2024 · DOI

TL;DR

Whombat is a new, open-source, browser-based tool designed to streamline the annotation of bioacoustic recordings for machine learning. It addresses the significant challenges in creating high-quality datasets for biodiversity monitoring by offering intuitive visualization, annotation, and review features, fostering an iterative workflow between human annotators and ML models.

What Problem Does This Paper Address?

Machine learning (ML) holds immense promise for scaling up biodiversity monitoring through the analysis of bioacoustic recordings. However, the effectiveness of these ML models hinges on the quality of the training and evaluation data. Creating these datasets is a complex, labor-intensive process fraught with challenges: managing vast collections of audio data with associated metadata, developing flexible tools to handle the diverse range of animal vocalizations, and the scarcity of expert annotators. Existing tools often fall short, lacking features crucial for efficient, collaborative, and iterative annotation workflows that are essential for building reliable ML models, especially for high-stakes applications like conservation.

Methods

Whombat is presented as a user-friendly, browser-based interface built with a server-client architecture. The backend is developed in Python, leveraging libraries like SciPy and NumPy for audio processing and FastAPI for its RESTful API. Data is stored in a relational database, with SQLite as the default, but configurable to other systems like MySQL or PostgreSQL. The frontend is built using TypeScript and React, providing a declarative and component-based UI.

Key features include:

  • Dataset Management: Importing and organizing audio recordings (WAV format) and their associated metadata.
  • Visualization and Exploration: Dynamic spectrogram generation for efficient navigation and identification of sound events, with configurable parameters.
  • Annotation Tools: Flexible annotation capabilities, including drawing temporal intervals or bounding boxes on spectrograms, and tagging with key-value pairs.
  • Quality Assurance: Tools for reviewing and managing annotations, flagging issues, and tracking progress.
  • User Training: A feature to train novice annotators using verified data, helping them develop identification skills.
  • ML Integration: The ability to import ML model predictions for comparison with annotations, facilitating an iterative feedback loop for model improvement.
  • Data Export: Exporting annotations in JSON (COCO format) or CSV for use in ML training or further analysis.

The tool is designed to be open-source and accessible, runnable on personal computers without coding skills, and can be self-hosted or accessed remotely.

Datasets

The paper describes two primary use cases that highlight the application of Whombat:

  1. UK Bat Call Identification: The Bat Conservation Trust (BCT) uses Whombat to annotate recordings for the BatDetect tool, aiming to improve bat detection and classification. This involves annotating 29 independent datasets of bat recordings, resulting in over 70,000 annotated calls.
  2. Bird Song Annotation in the Pacific Northwest: Researchers from Oregon State University and the USDA Forest Service are using Whombat to annotate avian sounds, with over 30,000 annotations collected. This data is used to validate ML predictions for monitoring species like the northern spotted owl and marbled murrelet, as well as supporting broader biodiversity monitoring across more than 80 species.

No new benchmarks or datasets were introduced as part of this paper; rather, Whombat is presented as a tool to facilitate the creation and management of such datasets.

Key Insights and Ideas

  • Bridging the Annotation-ML Gap: The core insight is that robust ML for bioacoustics is fundamentally limited by the availability of high-quality, well-managed annotated data. Whombat directly addresses this by providing a unified platform that integrates annotation with ML model development.
  • Iterative Improvement is Key: The tool's design emphasizes a feedback loop where ML predictions can inform and refine human annotations, and vice-versa. This iterative process is crucial for achieving high accuracy and reliability in bioacoustic ML models.
  • Flexibility is Paramount: Animal vocalizations are incredibly diverse. Whombat's flexible annotation system, configurable spectrograms, and support for rich metadata are essential for accommodating this diversity, unlike more specialized or generic tools.
  • Democratizing Bioacoustic ML: By being open-source, browser-based, and requiring no coding skills for basic use, Whombat lowers the barrier to entry for researchers and conservationists, enabling broader participation in ML-driven bioacoustic research.

Key Results

The paper showcases Whombat's utility through two distinct use cases:

  • Bat Call Annotation: The BCT has annotated over 70,000 bat calls using Whombat, with contributions from more than 15 annotators. This work is refining ML algorithms for bat detection and classification, aiding conservation efforts.
  • Bird Song Annotation: Over 30,000 avian sound annotations have been made using Whombat, supporting the monitoring of federally threatened species and broader biodiversity assessments in the Pacific Northwest. The tool's dynamic spectrogram adjustments and flexible annotation capabilities have improved annotation quality and efficiency.

Qualitatively, Whombat has demonstrated its ability to manage large datasets, facilitate collaborative work, and support an iterative ML development workflow.

Limitations and Future Work

The authors acknowledge that their evaluation of existing tools was subjective and may be biased by their experience. While Whombat aims to be comprehensive, they suggest several areas for future development:

  • Sequencing Annotations: The ability to group annotations into sequences would be beneficial for analyzing more complex sound patterns.
  • Model Comparison and Data Iteration Visualizations: Enhanced tools for visualizing the comparison of different ML models and the iterative refinement of data would further support the ML development workflow.
  • Dashboards for Ecological Insights: Developing dashboards that provide ecological insights directly from the annotated data would be valuable for conservation applications.
  • Citizen Science Integration: Exploring a citizen science approach by evaluating the reliability of user annotations could significantly expand the annotation capacity.
  • Embedded ML Detectors: While intentionally decoupled, the authors note that integrating ML detectors directly into the tool could offer further convenience for some users.

Summary updated

Abstract

Abstract Automated analysis of bioacoustic recordings using machine learning (ML) methods has the potential to greatly scale biodiversity monitoring efforts. The use of ML for high‐stakes applications, such as conservation and scientific research, demands a data‐centric approach with a focus on selecting and utilizing carefully annotated and curated evaluation and training data that are relevant and representative. Creating annotated bioacoustic datasets presents a number of challenges, such as managing large collections of recordings with associated metadata, developing flexible annotation tools that can accommodate the diverse range of vocalization profiles of different organisms and addressing the scarcity of expert annotators. We present Whombat, a user‐friendly, browser‐based interface for managing audio recordings and annotation projects, with several visualization, exploration and annotation tools. It enables users to quickly annotate, review, and share annotations, as well as visualize and evaluate a set of machine learning predictions on a dataset. The tool facilitates an iterative workflow where user annotations and machine learning predictions feedback to enhance model performance and annotation quality. We demonstrate the flexibility of Whombat by showcasing two distinct use cases: (1) a project aimed at enhancing automated UK bat call identification at the Bat Conservation Trust (BCT), and (2) a collaborative effort among the USDA Forest Service and Oregon State University researchers exploring bioacoustic applications and extending automated avian classification models in the Pacific Northwest, USA. Whombat is a flexible tool that can effectively address the challenges of annotation for bioacoustic research. It can be used for individual and collaborative work, hosted on a shared server or accessed remotely, or run on a personal computer without the need for coding skills. The code is open‐source, and we provide a user guide.
BibTeX
@inproceedings{balvanera2024whombat,
  author = {Santiago Martínez Balvanera and Oisin Mac Aodha and Matthew J. Weldy and Holly Pringle and Ella Browning and Kate E. Jones},
  title = {{Whombat: An open‐source audio annotation tool for machine learning assisted bioacoustics}},
  booktitle = {Methods},
  year = {2024},
  doi = {10.1111/2041-210x.14468},
}
Figure