Large-scale Speaker Retrieval on Random Speaker Variability Subspace

Suwon Shon, Younggun Lee, Taesu Kim

Abstract

This paper describes a fast speaker search system to retrieve segments of the same voice identity in large-scale data. While Locality Sensitive Hashing (LSH) enables quick retrieval in conjunction with i-vectors, its data-blind nature often leads to redundancy. We propose Random Speaker-variability Subspace (RSS) projection to map data into LSH-based hash tables. By projecting onto randomly generated speaker variability spaces rather than completely random subspaces, we increase the probability of same-speaker representations falling into the same hash bins. Experimental results demonstrate that the proposed approach is 100 times faster than linear search and 7 times faster than standard LSH while maintaining high accuracy.

Introduction

With the explosion of audio content on social media platforms, searching for specific voice identities has become a significant challenge. Applications range from copyright protection to the detection of deepfake voice cloning. The traditional approach, linear search, involves comparing a query embedding against every entry in a database, which is computationally prohibitive for millions of hours of audio.

Locality Sensitive Hashing (LSH) is a common alternative that uses random hyperplanes to "bucket" similar vectors. However, LSH is "data-blind" — it does not account for the underlying distribution of speaker characteristics. This leads to redundant hash bits and requires a large number of hash tables to maintain accuracy, which eventually degrades search speed.

Proposed Method: Random Speaker-variability Subspace (RSS)

To address the limitations of standard LSH, we introduce RSS projection. Unlike standard LSH, which uses purely random directions, RSS incorporates "weak supervision" by leveraging speaker-specific data from a training cohort.

The process involves:

  1. Selecting a random subset of speakers from a large training cohort (e.g., VoxCeleb).
  2. Applying Linear Discriminant Analysis (LDA) to this subset to identify dimensions that maximize inter-speaker variability.
  3. Using the resulting LDA matrix as the projection for a hash table.

By repeating this with different random subsets, we create multiple hash tables that are inherently better at distinguishing voices than purely random ones.

Comparison of (a) Random projection, which ignores data distribution, and (b) RSS projection, which guarantees maximum inter-speaker variability for selected speakers.

Experimental Evaluation

We evaluated the system using the VoxCeleb 1 and 2 datasets. A search space of over 100,000 utterances was constructed to simulate large-scale retrieval. We compared two types of embeddings: traditional i-vectors and deep neural network-based x-vectors.

Key Insights

  • X-vector Efficiency: X-vectors were found to be more "hashing-friendly" than i-vectors. Because x-vectors are trained discriminatively to tell speakers apart, they map into hash bins more efficiently, requiring fewer bits to achieve high separation.
  • Reduced Redundancy: By ensuring every hash bit focuses on speaker variability, RSS eliminates the redundancy found in standard LSH where many random bits do not contribute to speaker discrimination.

Results

The proposed RSS method significantly outperforms both linear search and standard LSH. As shown in the performance trade-off graph, the proposed method maintains over 95% accuracy even at a 100x speed-up relative to linear search.

Accuracy relative to linear search vs. speed-up. The proposed RSS method with x-vectors achieves significantly higher speed-up while maintaining high accuracy compared to LSH and Randomsampling LDA.

In direct comparison, RSS is approximately 7 times faster than standard LSH for the same level of accuracy. This efficiency allows for a smaller memory footprint, making it ideal for massive-scale applications.

Conclusion

The Random Speaker-variability Subspace projection offers a powerful way to optimize large-scale speaker retrieval. By replacing purely random hashing with weakly supervised projections, we achieve dramatic speed-ups with minimal accuracy loss. Future work will explore the system's robustness against synthesized voices and its applicability to other acoustic domains like music retrieval.

Summary updated

Abstract

This paper describes a fast speaker search system to retrieve segments of the same voice identity in the large-scale data. A recent study shows that Locality Sensitive Hashing (LSH) enables quick retrieval of a relevant voice in the large-scale data in conjunction with i-vector while maintaining accuracy. In this paper, we proposed Random Speaker-variability Subspace (RSS) projection to map a data into LSH based hash tables. We hypothesized that rather than projecting on completely random subspace without considering data, projecting on randomly generated speaker variability space would give more chance to put the same speaker representation into the same hash bins, so we can use less number of hash tables. Multiple RSS can be generated by randomly selecting a subset of speakers from a large speaker cohort. From the experimental result, the proposed approach shows 100 times and 7 times faster than the linear search and LSH, respectively
BibTeX
@inproceedings{shon2019large,
  author = {Suwon Shon and Younggun Lee and Taesu Kim},
  title = {{Large-scale Speaker Retrieval on Random Speaker Variability Subspace}},
  booktitle = {Interspeech},
  year = {2019},
  url = {https://openreview.net/forum?id=iVrw_HDqXU},
}

Loading related content...

Figure