TL;DR
This paper tackles the challenging task of speaker diarization for children's speech, which is notoriously difficult due to rapid phonetic variations and frequent overlaps. By incorporating children's speech into training data for Probabilistic Linear Discriminant Analysis (PLDA) models and employing a multi-PLDA fusion strategy, the authors achieve significant improvements in diarization error rates.
What Problem Does This Paper Address?
Speaker diarization, the process of identifying "who spoke when" in an audio recording, is a crucial preprocessing step for many speech technologies. While significant progress has been made for adult speech in controlled environments, children's speech presents unique challenges. Children's vocalizations are often spontaneous, leading to rapid and unpredictable phonetic changes that make it hard to extract stable speaker information. Furthermore, conversations involving children, especially with guardians, frequently feature long overlaps and fast speaker turns, exacerbating segmentation difficulties compared to more structured adult conversations like business meetings. Existing diarization systems, often trained on adult speech, struggle with this mismatch, leading to higher error rates. This paper aims to bridge this gap by developing more robust diarization systems for child-guardian interactions.
Methods
The core of this work lies in enhancing existing speaker diarization pipelines, specifically the JHU system used in the DiHARD 2018 challenge. The system utilizes an i-vector and an x-vector approach for speaker representation.
- Feature Extraction: Standard Mel-frequency cepstral coefficients (MFCCs) are extracted, along with their deltas and delta-deltas, and cepstral mean normalization is applied.
- Speaker Representation:
- i-vectors: A traditional approach that compresses speaker information into a low-dimensional "total variability space."
- x-vectors: A more recent approach using deep neural networks (specifically, time-delayed neural networks or TDNNs) to learn segment-level embeddings that capture discriminative speaker information.
- Probabilistic Linear Discriminant Analysis (PLDA): This generative model is used for scoring i-vectors and x-vectors. The key innovation here is a multi-PLDA approach:
- Finer Classification PLDA: Trained to distinguish between individual speakers, treating each unique speaker as a separate class.
- Coarser Classification PLDA: Trained to classify speakers into broader categories (e.g., "child," "female," "male"). This aims to capture high-level speaker characteristics.
- Fusion: The likelihood ratio scores from the finer and coarser PLDAs are combined using a weighted average to leverage the strengths of both models.
- Clustering: Agglomerative Hierarchical Clustering (AHC) is used to group segments based on the fused PLDA scores.
Datasets
- VoxCeleb1 & VoxCeleb2: Used for training the speaker feature extractors (i-vector and x-vector). These datasets primarily contain adult speech.
- CMU Kids Corpus: Used for training the PLDA models. This corpus contains children's speech.
- CSLU Kids' Speech Corpus: Also used for training the PLDA models, featuring spontaneous speech from children.
- SEEDLingS Corpus: Used as the test set. This corpus contains day-long recordings of very young children (6-18 months old) in their home environment, representing a significant age difference from the training data.
Key Insights and Ideas
- Children's Speech is a Game Changer: Simply adding children's speech data to the training set for PLDA models significantly improves diarization performance, especially for the x-vector system. This highlights the importance of domain-specific training data.
- Multi-PLDA Fusion Shows Promise: Combining a "finer" PLDA (speaker-specific) with a "coarser" PLDA (category-specific) through fusion yields results close to the best overall performance. This suggests that high-level speaker categorization can complement individual speaker discrimination.
- Balanced Training for Coarser PLDA Matters: Training the coarser PLDA with a balanced distribution of categories (e.g., "child," "female," "male") leads to better performance than using an unbalanced set, indicating that class balance is crucial for effective coarse-level modeling.
- Oracle Speech Segmentation is Necessary (for now): Due to the challenges of accurately detecting speech in noisy, child-centric environments, the authors relied on "oracle" speech activity detection (SAD). This means they used ground truth segmentation, which is a limitation for real-world deployment.
Key Results
- DER Reduction with Kids Data: Including children's speech in PLDA training reduced the Diarization Error Rate (DER) by 2.74% for the x-vector system, achieving a best overall DER of 33.15%. The i-vector system showed no improvement from this inclusion.
- Fusion Performance: The multi-PLDA fusion system achieved a DER of 33.29% (using balanced training for the coarser PLDA), which is very close to the best single-system performance.
- Balanced vs. Unbalanced Training: Balanced training for the coarser PLDA resulted in a lower DER compared to unbalanced training, both before and after fusion.
Limitations and Future Work
- Age Mismatch: A significant age difference exists between the children in the training data (5-11 years old) and the test data (6-18 months old). The authors suggest investigating the effect of children's age and exploring ways to address this gap.
- Coarser Classifier Improvement: The authors believe a better coarse-level classifier (e.g., a DNN) could further enhance the fusion approach.
- Parameter Tuning: The optimal weighting parameter ('a') for the fusion process needs to be iteratively tuned for different setups.
- Speech Activity Detection (SAD): The reliance on oracle SAD is a major limitation. Developing robust DNN-based SAD models that can effectively handle children's speech, including short or incomplete utterances and various non-speech events, is a critical area for future work.
- Complex Scenarios: The paper acknowledges the complexity of children's speech diarization due to simultaneous events and suggests leveraging this event information. Further work is needed on handling overlaps, noisy conditions, and improving the robustness of manual annotations.
Summary updated