Introduction
Speaker clustering—the process of grouping speech segments by individual identity—is a cornerstone of modern audio processing, essential for tasks like speaker diarization and large-scale speaker linking. However, traditional pipelines, typically relying on i-vector extraction and Probabilistic Linear Discriminant Analysis (PLDA) followed by Agglomerative Hierarchical Clustering (AHC), face significant hurdles. Specifically, AHC suffers from a quadratic computational bottleneck ($O(N^2)$) as dataset sizes grow, and PLDA's linear assumptions often fail in noisy, real-world environments. This paper introduces the Discrete Tied Variational Autoencoder (DTVAE), a novel architecture designed to address these efficiency and robustness gaps.
Methodology: The DTVAE Architecture
The proposed DTVAE replaces the traditional linear PLDA step with a non-linear neural network designed to perform initial, coarse clustering through discrete latent variables.
Discrete Latent Variables and Mutual Information
Unlike standard VAEs that map data to a continuous latent space, DTVAE employs a softmax layer to map voice features into discrete categories. To enhance robustness against environmental noise (e.g., background chatter, echoes), the authors incorporate a Mutual Information (MI) term into the loss function. By maximizing the MI between the input speech and the latent representations, the model learns to prioritize speaker-specific features over transient noise.
Two-Stage Clustering Workflow
For scenarios with an unknown number of speakers, DTVAE acts as a strategic filter. It first partitions the data into initial "coarse" groups. AHC is then performed only within these smaller subsets. This "divide and conquer" approach eliminates the need for a global $N \times N$ similarity matrix, drastically reducing the computational load.
Experimental Setup
The method was evaluated using the Voxceleb1 dataset, which contains over 145,000 utterances from 1,251 speakers. This dataset is particularly challenging due to its real-world nature, featuring varying recording equipment and significant ambient noise. A test subset of 43,733 utterances was used to evaluate performance in speaker linking applications.
Key Results and Discussion
The DTVAE method was benchmarked against a standard i-vector + PLDA + AHC baseline. The results demonstrate improvements in both efficiency and accuracy.
Computational Efficiency
As shown in the performance analysis, DTVAE significantly mitigates the computational bottleneck. On a dataset of 10,000 utterances, the method achieved a 32% reduction in elapsed time compared to the baseline. The time savings become increasingly pronounced as the data size scales, validating the efficiency of the two-stage partitioning approach.
Clustering Accuracy
DTVAE consistently outperformed the baseline in accuracy (ACC) across various configurations:
- Large-Scale Linking: In tests with 10,000 utterances, DTVAE reached 55.6% ACC, surpassing the baseline's 52.2%.
- Fixed Category Clustering: When the number of speakers is known, the improvements are even more stark. For a 3-speaker scenario, DTVAE achieved 93.3% ACC, compared to 86.6% for the baseline.
Conclusion and Future Work
DTVAE represents a significant step toward robust, scalable speaker clustering. By leveraging non-linear neural mappings and discrete latent spaces, it overcomes the limitations of traditional linear models and the $O(N^2)$ complexity of standard AHC. While accuracy still faces challenges as speaker counts increase, future work aims to evolve DTVAE into a fully end-to-end system, potentially eliminating the need for secondary clustering steps entirely.
Summary updated