TL;DR
This paper introduces CryCeleb, the first large-scale, anonymized dataset of infant cries labeled with individual infant identities, enabling speaker verification for newborns. They also hosted the CryCeleb 2023 challenge, which saw significant improvements over baseline models but highlighted the difficulty of infant cry verification compared to adult speech.
What Problem Does This Paper Address?
Identifying individual infants from their cries is crucial for applications in hospitals and households with multiple babies, especially for health monitoring and diagnostics. However, collecting and labeling such data is challenging due to the scarcity of resources with multiple recordings per infant. Existing datasets, like the Chillanto database, are limited in the number of recordings per infant, hindering the development of robust infant speaker verification systems. This paper addresses this gap by creating a new, comprehensive dataset and a corresponding challenge to spur research in this area.
Methods
The core of this work is the creation and release of the CryCeleb dataset. This dataset consists of manually segmented "expiration" segments from infant cries, excluding "inspiration" and "no cry" segments, as well as any cries containing personally identifiable information. The dataset was curated from recordings made within an hour of birth or upon hospital discharge.
For the CryCeleb 2023 challenge, the task was framed as a speaker verification problem: given two cry recordings, determine if they originated from the same infant. Participants were tasked with developing systems that output a similarity score for each pair of cries, with a higher score indicating a higher probability of the cries belonging to the same infant. Submissions were ranked using the Equal Error Rate (EER), a standard metric in speaker verification.
The paper also presents baseline models:
- A "naive" ECAPA-TDNN model pre-trained on the large adult speaker dataset VoxCeleb.
- The same ECAPA-TDNN model fine-tuned on the CryCeleb training data. This fine-tuning process aimed to adapt the model to the unique characteristics of infant cries.
Datasets
- CryCeleb Dataset: A newly introduced benchmark dataset comprising over 6 hours of cry sounds from 786 unique infants. It includes anonymized infant identities and metadata such as recording period (birth/discharge), duration, and file paths. A significant portion of infants have recordings from both birth and discharge periods, enabling verification across different time frames.
- VoxCeleb: Used for pre-training the baseline ECAPA-TDNN model.
Key Insights and Ideas
- Infant cry verification is significantly harder than adult speaker verification: Even the top-performing systems in the CryCeleb 2023 challenge achieved EERs far from state-of-the-art adult speaker verification, indicating substantial room for improvement.
- Transfer learning is crucial: Fine-tuning a model pre-trained on adult speech (VoxCeleb) on infant cries yielded much better results than training from scratch on the infant cry data alone. This highlights the importance of leveraging knowledge from related domains.
- The "birth vs. discharge" recording distinction matters: The dataset's design, including recordings from both periods, allows for evaluating a system's ability to verify an infant over time, simulating real-world scenarios where infants might be identified days apart.
- Careful dataset splitting and pair selection are important: The authors strategically selected dev and test infants based on initial embedding similarities to make the challenge more engaging and to account for the inherent difficulty of infant cry verification.
Key Results
- The naive VoxCeleb baseline achieved an EER of 37.9% on the dev set and 38.1% on the test set.
- The fine-tuned ECAPA-TDNN model significantly improved performance, achieving an EER of 27.8% on the dev set and 28.2% on the test set.
- The top-performing system in the CryCeleb 2023 challenge achieved an EER of 25.8%, still considerably high.
- Training an ECAPA-TDNN model from scratch on CryCeleb resulted in a 32.0% EER on the dev set and 31.8% on the test set, underscoring the benefit of transfer learning.
Limitations and Future Work
- High EERs: The paper acknowledges that the EERs, even for top systems, are high, suggesting the task remains challenging. This is partly attributed to potential changes in physiological characteristics between birth and discharge that can impact cry patterns.
- Future Work: The authors invite the community to continue working on this dataset and problem to further understand infant cry signals and improve verification performance. They suggest exploring extensions beyond the verification task, potentially to identify cry reasons or health pathologies. The open-sourcing of the dataset and baseline code is intended to facilitate this future research.
Summary updated