TL;DR
This paper introduces a novel Convolutional Neural Network (CNN) approach for automatically detecting "creaky voice" (vocal fry) in challenging emergency call recordings. The CNN system significantly outperforms existing methods, achieving a notable F1 score of 0.64 on this difficult dataset, suggesting its potential for analyzing caller emotional states or detecting voice disguise.
What Problem Does This Paper Address?
Creaky voice, a distinct phonation mode often described as "vocal fry," is a fascinating area of speech science. While it's known to be used for various communicative reasons—from signaling social status and emotions like boredom or sadness to marking utterance endings and turn-taking—its acoustic characteristics are highly variable. This variability, influenced by conversational context, speaker individuality, and even physiological factors, makes automatic detection a significant challenge. Existing creaky voice detection systems, while functional on clean speech, struggle with the degraded audio quality, background noise, and unique acoustic properties of real-world emergency calls. This limitation hinders the potential use of creaky voice analysis for critical applications like assessing caller distress or identifying voice disguise in high-stakes scenarios.
Methods
The core of this research is a custom-built Convolutional Neural Network (CNN) designed for creaky voice recognition. The CNN architecture features three alternating 2D convolution and pooling layers, followed by a fully connected layer and a softmax output layer. Key architectural choices include:
- Convolutional Layers: Employing filters of varying sizes (5x5, 3x3) to capture different spatial features in the spectrograms.
- Pooling Layers: Max-pooling to down-sample the signal and reduce dimensionality.
- Normalization and Activation: Batch normalization after pooling and leaky ReLU activation for improved training stability and non-linearity.
For feature extraction, the system uses 32 mel-scale log filter-bank outputs, a standard practice in speech processing, extracted with a 25ms window and 10ms frame shift. To address the limited availability of labeled emergency call data, the researchers also explored data augmentation techniques, including formant shifting and adding noise to simulate telephone distortion.
Datasets
The study utilizes two primary datasets:
- ECR30: A newly compiled dataset of 30 authentic Finnish emergency call recordings from the Kuopio Emergency Response Centre. This dataset is characterized by its challenging audio quality (8 kHz sampling frequency, unknown telephone types, background noise) and contains approximately 50 speakers (callers and operators).
- ACC30: A subset of the Aalto University DSP Course Conversation Corpus, comprising 30 Finnish speakers from laboratory-like conversational recordings. This dataset serves as a source of cleaner, out-of-domain speech for training and augmentation.
The ECR30 dataset was manually labeled for creaky voice using a combination of auditory observation, pitch contour analysis, and spectrogram examination, following established techniques.
Key Insights and Ideas
- Emergency Calls are Tough: The inherent noise, low sampling rates, and varied acoustic conditions of emergency calls make creaky voice detection significantly harder than on clean speech.
- Domain Mismatch Matters: Training a creaky voice detector on clean conversational speech and testing it on emergency calls leads to a substantial drop in performance, highlighting the importance of domain-specific training data.
- CNNs Show Promise: Despite the challenges, the proposed CNN architecture demonstrates a strong capability for creaky voice recognition, outperforming traditional methods on this difficult task.
- Data Augmentation Helps: Techniques like formant shifting and noise injection can improve the CNN's robustness and recall, though they might increase false positives.
- Creak and Emotion Link: The study's ANOVA results suggest that emergency contexts are associated with less creakiness compared to casual conversations, hinting at a potential link between stress and reduced creak.
Key Results
The paper presents a comparative analysis of their CNN system against the widely used Voice Analysis Toolkit (VAT).
- CNN vs. VAT on Emergency Calls: The CNN system achieved an F1 score of 0.64 on the ECR30 test set, a significant improvement over the VAT's F1 score of 0.41.
- Impact of Training Data:
- A CNN trained solely on clean ACC30 data and tested on ECR30 achieved an F1 of 0.52.
- When trained on a combination of ACC30 and some ECR26 emergency calls, the F1 rose to 0.61.
- With data augmentation applied to the combined training set, the F1 reached 0.64.
- ANOVA Findings: A two-way ANOVA revealed that the "emergency" context had a statistically significant effect on creak proportions (p < 0.001), with less creak observed in emergency calls. The effect of "sex" was not statistically significant on its own, but the interaction between "corpus" and "sex" was significant (p = 0.004), suggesting sex-related creak patterns might differ between emergency and casual contexts.
Limitations and Future Work
- Limited Data: The authors acknowledge that the amount of emergency call data available for training and testing is still limited, which impacts the ultimate performance ceiling.
- Manual Labeling Variability: Even trained phoneticians can have variations in creak boundary labeling, which affects the ground truth for evaluating automatic systems.
- Future Directions:
- More Data: The primary goal is to collect and label more authentic emergency call recordings to further improve the CNN model.
- Exploratory Recognizer: The current exploratory creak recognizer can be used to generate more labeled data.
- Deeper Analysis: Investigating the precise relationship between creaky voice and caller emotional states in emergency situations.
- Broader Applications: Exploring the use of DNN-based creaky voice detection for other forensic applications beyond emergency calls.
Summary updated