Abstract
This paper describes a fast speaker search system to retrieve segments of the same voice identity in large-scale data. While Locality Sensitive Hashing (LSH) enables quick retrieval in conjunction with i-vectors, its data-blind nature often leads to redundancy. We propose Random Speaker-variability Subspace (RSS) projection to map data into LSH-based hash tables. By projecting onto randomly generated speaker variability spaces rather than completely random subspaces, we increase the probability of same-speaker representations falling into the same hash bins. Experimental results demonstrate that the proposed approach is 100 times faster than linear search and 7 times faster than standard LSH while maintaining high accuracy.
Introduction
With the explosion of audio content on social media platforms, searching for specific voice identities has become a significant challenge. Applications range from copyright protection to the detection of deepfake voice cloning. The traditional approach, linear search, involves comparing a query embedding against every entry in a database, which is computationally prohibitive for millions of hours of audio.
Locality Sensitive Hashing (LSH) is a common alternative that uses random hyperplanes to "bucket" similar vectors. However, LSH is "data-blind" — it does not account for the underlying distribution of speaker characteristics. This leads to redundant hash bits and requires a large number of hash tables to maintain accuracy, which eventually degrades search speed.
Proposed Method: Random Speaker-variability Subspace (RSS)
To address the limitations of standard LSH, we introduce RSS projection. Unlike standard LSH, which uses purely random directions, RSS incorporates "weak supervision" by leveraging speaker-specific data from a training cohort.
The process involves:
- Selecting a random subset of speakers from a large training cohort (e.g., VoxCeleb).
- Applying Linear Discriminant Analysis (LDA) to this subset to identify dimensions that maximize inter-speaker variability.
- Using the resulting LDA matrix as the projection for a hash table.
By repeating this with different random subsets, we create multiple hash tables that are inherently better at distinguishing voices than purely random ones.
Experimental Evaluation
We evaluated the system using the VoxCeleb 1 and 2 datasets. A search space of over 100,000 utterances was constructed to simulate large-scale retrieval. We compared two types of embeddings: traditional i-vectors and deep neural network-based x-vectors.
Key Insights
- X-vector Efficiency: X-vectors were found to be more "hashing-friendly" than i-vectors. Because x-vectors are trained discriminatively to tell speakers apart, they map into hash bins more efficiently, requiring fewer bits to achieve high separation.
- Reduced Redundancy: By ensuring every hash bit focuses on speaker variability, RSS eliminates the redundancy found in standard LSH where many random bits do not contribute to speaker discrimination.
Results
The proposed RSS method significantly outperforms both linear search and standard LSH. As shown in the performance trade-off graph, the proposed method maintains over 95% accuracy even at a 100x speed-up relative to linear search.
In direct comparison, RSS is approximately 7 times faster than standard LSH for the same level of accuracy. This efficiency allows for a smaller memory footprint, making it ideal for massive-scale applications.
Conclusion
The Random Speaker-variability Subspace projection offers a powerful way to optimize large-scale speaker retrieval. By replacing purely random hashing with weakly supervised projections, we achieve dramatic speed-ups with minimal accuracy loss. Future work will explore the system's robustness against synthesized voices and its applicability to other acoustic domains like music retrieval.
Summary updated