TL;DR
This paper tackles the challenge of identifying specific speakers in mono-channel recordings, common in real-world scenarios like criminal investigations. They propose novel methods for automatically creating speaker enrollment models from these mixed-voice recordings and evaluate their effectiveness against existing techniques. The findings suggest that even simpler methods can perform well, offering practical solutions for resource-constrained environments.
What Problem Does This Paper Address?
In many real-world applications, audio recordings are not neatly separated by speaker. This is particularly true for mono-channel telephony recordings, where the speech from all participants is mixed into a single audio stream. This poses a significant challenge for speaker recognition systems, which typically rely on clean, single-speaker audio for both enrollment (creating a speaker's voice profile) and verification (checking if a speaker is present in a new recording).
The authors focus on investigative scenarios where law enforcement agencies have access to multiple mono-channel conversations involving a person of interest. The goal is to build a reliable speaker model for this individual using these mixed recordings and then use that model to find instances of the target speaker in other, similar recordings. A key difficulty is that the enrollment recordings likely contain not only the target speaker but also their conversation partners. If these partners also appear in the test recordings, they could be mistakenly identified as the target speaker, leading to false positives. Existing methods often require manually labeled "assist segments" or clean enrollment data, which are not available in this context.
Methods
The paper proposes several approaches for automatically creating speaker enrollment models from multi-speaker mono-channel recordings. These methods are designed to work with x-vectors, which are DNN-based speaker embeddings extracted from short audio segments.
Diarization: Before enrollment model creation and for the verification phase, the system needs to identify which speaker is speaking at any given time. The authors explore two diarization methods:
- PCA-based diarization: A simple method that projects x-vectors onto their principal component. Segments with projections above a threshold are assigned to one speaker, and others to a second speaker. This method inherently assumes exactly two speakers.
- VBx diarization: A more sophisticated, state-of-the-art method based on Variational Bayes and PLDA. The authors adapt it to constrain the number of detected speakers to two.
Enrollment Model Creation: * Median Embedding: This is the simplest approach, where the enrollment model is the element-wise median of all x-vectors belonging to the target speaker across all enrollment conversations. The median is preferred over the mean as it's less sensitive to outliers (i.e., the conversation partners' speech). * Eigenvectors Intersection: Inspired by PCA diarization, this method assumes the main eigenvector of each conversation represents the direction of speaker variability. The intersection of these eigenvectors from multiple conversations is hypothesized to point towards the target speaker's embedding space. * Complete Cluster Search: This method first diarizes each enrollment call, assuming one of the two detected speakers is the target. It then explores all possible combinations of assigning one speaker from each call to the target speaker. Each combination is evaluated using either the standard deviation of the embeddings or a PLDA likelihood score, with the best combination forming the enrollment model. This method is computationally intensive as the number of combinations grows exponentially with the number of enrollment calls.
Verification: The verification process involves diarizing the test mono-channel conversation into two speakers. The x-vectors from each detected speaker are then scored against the enrollment model. The maximum score is used as the final verification score. The authors provide a theoretical justification for this "max" approach, showing it approximates the log-likelihood ratio (LLR) under certain assumptions.
Datasets
- Roxanne Simulated Data (ROXSD): This dataset was collected to aid the development of tools for criminal investigations. It contains partially scripted calls between members of simulated criminal networks, inspired by real cases. The authors used the first eight chronologically recorded calls of each of 87 unique speakers for enrollment and the remaining calls for testing.
- Real Case Data: Conversations obtained from a real criminal investigation were also used. This database consisted of approximately 200 mono conversations with varied lengths, labeled with speaker PIN codes. For evaluation, 25 speakers with at least three conversations were selected, using a one-leave-out cross-validation approach.
Key Insights and Ideas
- The "Max" Score for Verification is a Reasonable Approximation: The paper provides theoretical backing for using the maximum score from diarized segments as the verification score, simplifying the process and aligning with practical implementations.
- Simpler Enrollment Methods Can Be Effective: The "Median Embedding" method, despite its simplicity, showed competitive results, especially when a limited number of enrollment calls are available. This is crucial for scenarios where extensive data collection is difficult.
- PCA Diarization Outperforms VBx in This Context: Surprisingly, the simpler PCA-based diarization method yielded better results than the more complex VBx method in their experiments. The authors attribute this to VBx's many tunable parameters, which were not extensively explored.
- Preprocessing Matters: Applying preprocessing (like LDA projection and mean subtraction) before averaging embeddings for the enrollment model (the "pp-mean" approach) generally led to better performance than averaging first and then preprocessing ("mean-pp").
- t-norm Significantly Improves Performance on Real Data: For cosine similarity scoring on real case data, t-norm normalization provided a substantial performance boost, highlighting its importance in mitigating biases in multi-speaker test recordings.
Key Results
- Best Performance: The "Complete Cluster Search" method with PLDA likelihood objective and PCA diarization, with preprocessing applied before averaging ("pp-mean"), generally achieved the best Equal Error Rates (EER) on the ROXSD dataset. For example, EERs as low as 3.61% were reported.
- Impact of Enrollment Calls: The median method's performance could be somewhat variable, but it was a good choice when only a few enrollment calls were available. The intersection and cluster methods showed more consistent performance as the number of enrollment calls increased.
- Real Case Performance: On the real case data, the "Intersection" enrollment method with PCA verification and t-norm normalization showed strong performance, with EERs around 9.0%. The paper also noted that t-norm significantly improved performance compared to not using it.
Limitations and Future Work
- Limited Number of Speakers in Diarization: The current methods are primarily designed for scenarios with exactly two speakers per conversation. Extending these to handle more than two speakers in a single channel is a significant area for future research.
- Tuning of VBx Parameters: The authors acknowledge that the VBx diarization method has many tunable parameters that were not fully explored, which might have contributed to its underperformance compared to PCA diarization.
- Database Size: While the paper uses both simulated and real-world data, the authors suggest that larger databases would be beneficial to further confirm their findings.
- More Refined Embedding Averaging: Future work could explore more sophisticated schemes for averaging embeddings beyond simple median or mean, potentially leading to improved enrollment models.
Summary updated