Extended performance analysis of deep-learning algorithms for mice vocalization segmentation

Daniele Baggi, Marika Premoli, Alessandro Gnutti, Sara Anna Bonini, Riccardo Leonardi, Maurizio Memo, Pierangelo MiglioratiScientific 2023 · DOI

TL;DR

This paper presents a comprehensive performance analysis of three deep learning architectures (AE, UNET, RNN) for segmenting ultrasonic vocalizations (USVs) in mice. The authors introduce a new dataset and demonstrate that UNET and AE achieve state-of-the-art performance, outperforming existing methods and offering a valuable benchmark for future research in animal communication analysis.

What Problem Does This Paper Address?

Understanding animal communication is crucial for ethological studies and for investigating neurological and neuropharmacological conditions in mice. Ultrasonic vocalizations (USVs) are a key component of this communication, particularly in mice, serving roles in maternal care, social interaction, and courtship. While automated systems for detecting and classifying USVs exist, the accuracy of these systems heavily relies on precise USV segmentation – accurately identifying the start and end times of each vocalization. Manual segmentation is time-consuming and prone to operator variability. Existing automated methods, while improving, haven't been thoroughly evaluated using the latest deep learning techniques for this critical segmentation step. This paper aims to fill this gap by rigorously comparing the performance of three advanced deep learning models for USV segmentation.

Methods

The researchers investigated three supervised deep learning architectures for USV segmentation:

  • Auto-Encoder (AE): A neural network designed for learning efficient data representations. In this context, it's used to encode spectrograms into a compressed latent space and then decode them to reconstruct the segmentation mask.
  • U-NET Neural Network (UNET): A convolutional neural network architecture originally developed for biomedical image segmentation. It features a U-shaped structure with skip connections that allow information from earlier layers to be passed directly to later layers, aiding in precise localization.
  • Recurrent Neural Network (RNN): A type of neural network well-suited for sequential data, like time series. Here, it's used to capture temporal dependencies within the spectrograms.

All models take spectrograms of mouse vocalizations as input and output a binary vector indicating the presence or absence of a USV for each time frame. To handle computational costs, spectrograms were divided into overlapping segments. A post-processing step using mathematical morphology (closing and opening operations) was applied to refine the segmentation outputs, correcting for noisy spikes and small interruptions.

Datasets

  • Custom Dataset: The authors created a new dataset by recording audio tracks from 40 control and 39 NF-κB p50 knock-out pups. This dataset consists of 198 three-minute audio tracks, from which spectrograms were generated. Experts manually segmented these spectrograms, resulting in ground truth labels for 49,750 USVs. This dataset was split into training (80%) and testing (20%) sets.
  • External Dataset: The performance of the models was also evaluated on a publicly available dataset from a previous study (doi.org/10.5281/zenodo.3428024), which includes audio tracks from different mouse strains and contexts (maternal isolation, courtship) to assess the models' generalizability.

Key Insights and Ideas

  • Segmentation is Key: The paper strongly emphasizes that accurate USV segmentation is the foundational step for any subsequent analysis or classification of vocalizations.
  • Deep Learning Dominance: The study highlights the superior performance of deep learning models (AE and UNET) over traditional signal processing methods for USV segmentation.
  • UNET's Versatility: The UNET architecture, known for its success in image segmentation, proves highly effective for spectrogram-based USV segmentation, demonstrating strong performance across different datasets.
  • Post-processing Matters: While the deep learning models achieve high accuracy, a post-processing step using mathematical morphology significantly improves segmentation quality by cleaning up false positives and filling gaps.
  • Parameter Efficiency: Notably, the best-performing models (AE and UNET) are also relatively parameter-efficient, making them computationally feasible for broader application.

Key Results

  • Time Frame Classification: All three proposed deep learning models achieved precision and recall scores exceeding 90% for time frame classification. AE and UNET surpassed 95%.
  • USV Segmentation: For USV segmentation, AE and UNET achieved precision and recall rates exceeding 90%, with UNET showing the highest performance. RNN performed well but slightly lower than AE and UNET.
  • Comparison to State-of-the-Art: The proposed models, particularly UNET and AE, outperformed established software tools like DeepSqueak, HybridMouse, A-MUD, and USVSEG in terms of precision and recall on their custom dataset.
  • External Dataset Performance: UNET maintained its leading performance on the external dataset, demonstrating robustness and generalizability.
  • Computational Efficiency: The AE and UNET models are characterized by a relatively small number of parameters, leading to efficient computation. For example, segmenting a 3-minute audio track took approximately 25 seconds for AE and UNET.

Limitations and Future Work

  • Post-processing Optimization: While post-processing improved results, the optimal kernel sizes for this step were determined through extensive testing and might require further fine-tuning for diverse datasets.
  • External Dataset Variability: Performance on the external dataset showed a slight decrease compared to the custom dataset, attributed to variations in signal-to-noise ratio and potentially different recording procedures. This highlights the ongoing challenge of achieving consistent performance across highly diverse datasets.
  • Future Integration: The authors plan to integrate their segmentation models into a full USV segmentation-classification system. This integration is expected to reveal how improved segmentation accuracy influences classification outcomes.
  • Broader Applicability: Further validation on a wider range of strains and experimental conditions is suggested to fully assess the models' applicability.

Summary updated

Abstract

Ultrasonic vocalizations (USVs) analysis represents a fundamental tool to study animal communication. It can be used to perform a behavioral investigation of mice for ethological studies and in the field of neuroscience and neuropharmacology. The USVs are usually recorded with a microphone sensitive to ultrasound frequencies and then processed by specific software, which help the operator to identify and characterize different families of calls. Recently, many automated systems have been proposed for automatically performing both the detection and the classification of the USVs. Of course, the USV segmentation represents the crucial step for the general framework, since the quality of the call processing strictly depends on how accurately the call itself has been previously detected. In this paper, we investigate the performance of three supervised deep learning methods for automated USV segmentation: an Auto-Encoder Neural Network (AE), a U-NET Neural Network (UNET) and a Recurrent Neural Network (RNN). The proposed models receive as input the spectrogram associated with the recorded audio track and return as output the regions in which the USV calls have been detected. To evaluate the performance of the models, we have built a dataset by recording several audio tracks and manually segmenting the corresponding USV spectrograms generated with the Avisoft software, producing in this way the ground-truth (GT) used for training. All three proposed architectures demonstrated precision and recall scores exceeding [Formula: see text], with UNET and AE achieving values above [Formula: see text], surpassing other state-of-the-art methods that were considered for comparison in this study. Additionally, the evaluation was extended to an external dataset, where UNET once again exhibited the highest performance. We suggest that our experimental results may represent a valuable benchmark for future works.
BibTeX
@inproceedings{baggi2023extended,
  author = {Daniele Baggi and Marika Premoli and Alessandro Gnutti and Sara Anna Bonini and Riccardo Leonardi and Maurizio Memo and Pierangelo Migliorati},
  title = {{Extended performance analysis of deep-learning algorithms for mice vocalization segmentation}},
  booktitle = {Scientific},
  year = {2023},
  doi = {10.1038/s41598-023-38186-7},
}
Figure