On-Bird Sound Recordings: Automatic Acoustic Recognition of Activities and Contexts

Dan Stowell, Emmanouil Benetos, Lisa F. Gill

TL;DR

Researchers have developed a way to automatically "read" the daily lives of birds by analyzing audio from tiny microphone backpacks. By comparing machine learning classifiers and event-detection algorithms, the study demonstrates that we can identify specific behaviors like flying, walking, and vocalizing, though individual differences between birds remain a significant hurdle for universal automation.

What Problem Does This Paper Address?

In the world of animal behavior research, "bio-logging" (using GPS or accelerometers) has revolutionized our understanding of where animals go. However, these sensors often fail to capture the context of behavior. A GPS coordinate doesn't tell you if a bird is in a noisy colony, being harassed by a predator, or interacting with a mate.

While audio recording provides this rich context, it creates a massive data bottleneck. Manually annotating hundreds of hours of "on-bird" audio is practically impossible, especially since researchers can't always see what a free-flying bird is doing to verify the sounds. This paper tackles the challenge of Automatic Acoustic Recognition, attempting to turn raw audio into a "time budget" of an animal's activities and environment.

Methods

The authors compared two distinct paradigms for sound analysis:

  1. The Classifier-based System: This treats audio clips (5-second segments) as "scenes." It uses a technique called spherical k-means feature learning to transform Mel-spectrograms into a rich feature space, which is then fed into a Random Forest classifier. This approach is designed to be scalable and robust to "big data."
  2. The Event Detection System: This uses Probabilistic Latent Component Analysis (PLCA). Instead of classifying a whole segment, it tries to decompose the spectrogram into a series of overlapping sound "exemplars." It’s essentially trying to find specific "events" (like a single wing flap) within the stream.

To refine the results, the team applied Hidden Markov Models (HMMs) as a post-processing step. The HMMs use the "grammar" of bird behavior (e.g., a bird is likely to keep flying if it was flying a second ago) to smooth out erratic predictions.

Datasets

The study utilized a unique, high-quality dataset of Eurasian jackdaws (Corvus monedula):

  • Captive Dataset: 8.4 hours of audio from 7 birds, meticulously video-validated. This means researchers watched video footage to ensure that when the computer heard "walking," the bird was actually walking.
  • Field Dataset: 18.5 hours of audio from 6 free-living birds, human-coded based on the "acoustic signatures" learned from the captive study.
  • Categories: The labels included movements (flying, walking, shaking), vocalizations (focal calls, background calls), and environmental contexts (nest, colony noise).

Key Insights and Ideas

  • The "Individual" Problem: One of the most significant findings was that models trained on one bird didn't always generalize well to another. Differences in how a backpack fits or how an individual bird moves create unique "acoustic fingerprints" that can confuse a general model.
  • Vocalizations vs. Movements: The classifier was particularly good at identifying the "focal call" (the voice of the bird wearing the mic), likely because the microphone placement ensures a very high signal-to-noise ratio for the bearer's own voice.
  • Scene vs. Event: The classifier-based "scene" approach generally outperformed the PLCA "event" approach. This suggests that for biological time-budgeting, identifying the general state of the bird over a few seconds is more reliable than trying to pinpoint every micro-event.
  • The HMM Trade-off: Adding HMM post-processing improved the F-score (a balance of precision and recall) by smoothing out "flickering" predictions, but it actually lowered the AUC (Area Under the Curve), indicating that while it makes the output look cleaner, it might occasionally suppress real, brief transitions.

Key Results

  • Performance Lead: The classifier-based system consistently outperformed the PLCA system by an average of 5 percentage points in AUC and 8 percentage points in F-score.
  • Generalization Penalty: Moving from "per-individual" training to "pooled" training (testing on a bird the model hadn't seen before) caused a substantial drop in recognition quality.
  • Top Categories: The strongest results were achieved for focal calls, flying, and self-maintenance (preening). These activities have distinct, repetitive acoustic patterns that are easier for the algorithms to "grip."

Limitations and Future Work

The authors are transparent about the fact that we aren't at "plug-and-play" automation yet. The current systems require at least some manual annotation of a new bird's data to achieve high accuracy.

Future directions include: * Dynamic Time Resolution: Instead of fixed 5-second windows, using different timescales for different behaviors (e.g., a "shaking" event is much shorter than a "flying" state). * Multi-Sensor Fusion: Combining this acoustic data with accelerometer data to resolve ambiguities (e.g., distinguishing between a bird ruffling its feathers and the sound of wind hitting the microphone). * Improved Generalization: Developing features that are invariant to the specific fit of a backpack or the unique gait of an individual bird.

Summary updated

Abstract

We introduce a novel approach to studying animal behavior and the context in which it occurs, through the use of microphone backpacks carried on the backs of individual free-flying birds. These sensors are increasingly used by animal behavior researchers to study individual vocalizations of freely behaving animals, even in the field. However, such devices may record more than an animal's vocal behavior, and have the potential to be used for investigating specific activities (movement) and context (background) within which vocalizations occur. To facilitate this approach, we investigate the automatic annotation of such recordings through two different sound scene analysis paradigms: A scene-classification method using feature learning, and an event-detection method using probabilistic latent component analysis. We analyze recordings made with Eurasian jackdaws (Corvus monedula) in both captive and field settings. Results are comparable with the state of the art in sound scene analysis; we find that the current recognition quality level enables scalable automatic annotation of audio logger data, given partial annotation, but also find that individual differences between animals and/or their backpacks limit the generalization from one individual to another. we consider the interrelation of “scenes” and “events” in this particular task, and issues of temporal resolution.
BibTeX
@article{stowell2017bird,
  author = {Dan Stowell and Emmanouil Benetos and Lisa F. Gill},
  title = {{On-Bird Sound Recordings: Automatic Acoustic Recognition of Activities and Contexts}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2017},
  url = {https://openreview.net/forum?id=iweBsTXdoO},
}

Loading related content...

Figure