Abstract
This paper describes a fast speaker search system to retrieve segments of the same voice identity in large-scale data. While Locality Sensitive Hashing (LSH) enables quick retrieval, it often requires numerous hash tables to maintain accuracy. We propose Random Speaker-variability Subspace (RSS) projection to map data into LSH-based hash tables. By projecting onto randomly generated speaker variability spaces rather than completely random subspaces, we increase the probability of matching speaker representations landing in the same hash bins. Experimental results demonstrate that the proposed approach is 100 times faster than linear search and 7 times faster than standard LSH.
Introduction
The explosion of audio-visual content on platforms like YouTube has made searching for specific voice identities a significant technical challenge. Applications range from copyright protection and forensic analysis to defending against deepfakes. Standard Locality Sensitive Hashing (LSH) projects high-dimensional embeddings (i-vectors or x-vectors) into random "bins." However, standard LSH is often inefficient, requiring hundreds of hash tables to maintain accuracy, which increases memory usage and search latency.
Methods: Random Speaker-variability Subspace (RSS)
The proposed RSS projection introduces "weak supervision" into the hashing process. Instead of purely random mathematical projections, the system utilizes Linear Discriminant Analysis (LDA) on a randomly selected subset of speakers (a "cohort") from a larger training set.
As shown in the figure above, while standard LSH (a) uses redundant projections without considering data distribution, RSS (b) chooses directions that specifically maximize variability between different speakers. This ensures that utterances from the same speaker are more likely to land in the same hash bin. By repeating this process with different random subsets, multiple diverse yet "intelligent" hash tables are created.
Experimental Setup and Datasets
The study utilized the VoxCeleb 1 and 2 datasets, comprising over 1.2 million utterances from 7,365 speakers. The system was evaluated on two tasks:
- Speaker Identification: Matching a query to a known gallery (closed-set).
- Speaker Retrieval: Finding all instances of a target speaker within a pool of 100,000+ segments.
Results and Discussion
The RSS approach achieved a 100x speedup over linear search and a 7x speedup over traditional LSH while maintaining over 95% of baseline accuracy.
The efficiency of the method is reflected in the Hamming distance analysis. As shown in the graph, the proposed method creates a clearer "gap" (lower ratio) between same-speaker and different-speaker pairs in the hashed space compared to standard LSH and random-sampling LDA.
Furthermore, the study found that x-vectors are more "hash-friendly" than i-vectors. Because x-vectors are trained discriminatively via neural networks, they map into hash bins more efficiently than the Gaussian-assumed i-vectors. RSS also significantly reduces bit redundancy, allowing for shorter hash codes and fewer tables.
Conclusion
RSS projection offers a highly efficient solution for large-scale speaker retrieval by leveraging speaker variability during the hashing process. Future work may explore performance against synthesized speech and cross-domain robustness in noisy environments.
Summary updated