TL;DR
This paper tackles the challenge of speaker verification (SV) when speech is whispered, a mode that significantly alters acoustic features compared to normal speech. They propose a novel feature mapping technique that leverages cosine similarity in i-vector space to bridge this gap, achieving a substantial ~24% relative reduction in error rates for whispered speech SV while maintaining performance for neutral speech.
What Problem Does This Paper Address?
Speaker verification (SV) systems are designed to confirm a person's identity based on their voice. While these systems perform well with standard, "neutral" speech, their performance degrades significantly when encountering whispered speech. This degradation is due to fundamental differences in how whispered speech is produced: it lacks vocal cord vibration (voicing), has altered formant frequencies, and often involves hyper-articulation for intelligibility. This poses a problem for security applications (e.g., PIN entry) and forensic analysis, where whispered speech might be used intentionally or be the only mode of communication for some individuals. Existing methods often struggle to map whispered speech features to a representation that aligns with neutral speech i-vectors, leading to poor verification accuracy.
Methods
The core of this work is a novel feature mapping (FM) technique designed to transform whispered speech features into a representation that is more compatible with neutral speech i-vectors.
- Standard i-vector Framework: The paper builds upon the established i-vector framework for speaker verification. This involves extracting features (MFCCs), modeling them with a Gaussian Mixture Model (GMM) to train a Universal Background Model (UBM), and then learning a low-rank i-vector space (T-matrix) that captures speaker and channel variability. i-vectors are then extracted for enrollment and testing.
- Feature Mapping (FM): The key innovation lies in how the FM is trained. Instead of minimizing the Mean Squared Error (MSE) between whispered and neutral speech features (a common approach in prior work), the authors propose maximizing the cosine similarity between the i-vectors derived from mapped whispered speech and the i-vectors from neutral speech. This is done in the i-vector space itself.
- Objective Function: The proposed objective function aims to learn the FM parameters by maximizing the cosine similarity: $\psi(w_n, w') = \frac{w_n^T w'}{||w_n|| ||w' ||}$, where $w_n$ is the i-vector from neutral speech and $w'$ is the i-vector from the mapped whispered speech. The gradient of this function is used for training.
- Affine Transformation: The paper explores an affine transformation ($y = Ax + b$) for the feature mapping. They investigate both unconstrained and identity-constrained affine transformations, finding that the unconstrained version, particularly the bias term ($b$), plays a crucial role.
Datasets
The study utilizes a combination of publicly available and in-house datasets:
- CHAINS corpus: Contains neutral and whispered speech from 36 speakers.
- wTIMIT: Whisper version of the TIMIT corpus, with neutral and whispered speech from 48 speakers.
- VoxCeleb1: A large-scale audio-visual dataset with 1251 speakers.
- TIMIT data set: Standard dataset with 630 speakers.
- wSPIRE (in-house): Data from 9 speakers in two Indian dialect regions, recorded in both neutral and whispered modes.
All corpora were resampled to a common sampling frequency of 16kHz.
Key Insights and Ideas
- Cosine Similarity in i-vector Space is Key: The most significant insight is that directly optimizing for cosine similarity in the i-vector space is more effective for whispered SV than minimizing MSE in the feature space. This suggests that preserving speaker-specific information is better achieved by focusing on the latent representation (i-vectors) rather than raw acoustic features.
- Bias Term Captures Glottal Flow: The learned bias term ($b$) in the affine transformation shows a strong resemblance to glottal flow information. This is crucial because glottal flow is largely absent in whispered speech, and adding this bias effectively injects missing source information, significantly improving performance.
- Neutral SV Performance Maintained: A critical finding is that the proposed FM approach improves whispered SV performance without degrading the performance on neutral speech. This is a significant advantage over some baseline methods.
- Affine Transform Outperforms DNN: For feature mapping, the affine transformation with the proposed cosine similarity objective performed better than DNN-based mapping, potentially due to DNNs overfitting on specific training data channels.
Key Results
- ~24% Relative EER Reduction: The proposed method achieved a relative reduction of approximately 24% in the Equal Error Rate (EER) for whispered speech speaker verification compared to the best baseline.
- Improved Whispered SV: The EER for whispered speech was significantly reduced across various experimental conditions. For instance, with the affine transform and the proposed FM (FMcs), EERwh dropped to 15.12% and 16.24% in different configurations, compared to baseline WFM EERwh of 22.7% and 24.7%.
- Neutral SV Performance Unchanged: The EER for neutral speech (EERne) remained largely unchanged, typically around 4.12% or 5.31%, demonstrating the method's ability to preserve neutral SV performance.
- Affine Transform Effectiveness: The affine transform-based FM (FMcs with Af) consistently outperformed DNN-based FM (FMcs with DNN) and baseline methods.
Limitations and Future Work
- Limitations: The paper notes that while the proposed method improves whispered SV, the EER for whispered speech is still higher than for neutral speech. The authors also mention that retraining the T-matrix doesn't always further improve EERwh significantly when using their FM.
- Future Work: The authors suggest exploring joint learning of the T-matrix and the feature mapping for whispered SV. They also propose investigating the cosine similarity objective function with other feature types and extending the approach to other related tasks.
Summary updated