A Robust Speaker Clustering Method Based on Discrete Tied Variational Autoencoder

Chen Feng, Jianzong Wang, Tongxu Li, Junqing Peng, Jing Xiao

Introduction

Speaker clustering—the process of grouping speech segments by individual identity—is a cornerstone of modern audio processing, essential for tasks like speaker diarization and large-scale speaker linking. However, traditional pipelines, typically relying on i-vector extraction and Probabilistic Linear Discriminant Analysis (PLDA) followed by Agglomerative Hierarchical Clustering (AHC), face significant hurdles. Specifically, AHC suffers from a quadratic computational bottleneck ($O(N^2)$) as dataset sizes grow, and PLDA's linear assumptions often fail in noisy, real-world environments. This paper introduces the Discrete Tied Variational Autoencoder (DTVAE), a novel architecture designed to address these efficiency and robustness gaps.

Methodology: The DTVAE Architecture

The proposed DTVAE replaces the traditional linear PLDA step with a non-linear neural network designed to perform initial, coarse clustering through discrete latent variables.

The DTVAE architecture utilizes discrete latent variables and a two-stage workflow to optimize speaker clustering.

Discrete Latent Variables and Mutual Information

Unlike standard VAEs that map data to a continuous latent space, DTVAE employs a softmax layer to map voice features into discrete categories. To enhance robustness against environmental noise (e.g., background chatter, echoes), the authors incorporate a Mutual Information (MI) term into the loss function. By maximizing the MI between the input speech and the latent representations, the model learns to prioritize speaker-specific features over transient noise.

Two-Stage Clustering Workflow

For scenarios with an unknown number of speakers, DTVAE acts as a strategic filter. It first partitions the data into initial "coarse" groups. AHC is then performed only within these smaller subsets. This "divide and conquer" approach eliminates the need for a global $N \times N$ similarity matrix, drastically reducing the computational load.

Experimental Setup

The method was evaluated using the Voxceleb1 dataset, which contains over 145,000 utterances from 1,251 speakers. This dataset is particularly challenging due to its real-world nature, featuring varying recording equipment and significant ambient noise. A test subset of 43,733 utterances was used to evaluate performance in speaker linking applications.

Key Results and Discussion

The DTVAE method was benchmarked against a standard i-vector + PLDA + AHC baseline. The results demonstrate improvements in both efficiency and accuracy.

Comparison of processing time between the baseline and DTVAE as data size increases, showing a significant reduction in computational overhead.

Computational Efficiency

As shown in the performance analysis, DTVAE significantly mitigates the computational bottleneck. On a dataset of 10,000 utterances, the method achieved a 32% reduction in elapsed time compared to the baseline. The time savings become increasingly pronounced as the data size scales, validating the efficiency of the two-stage partitioning approach.

Clustering Accuracy

DTVAE consistently outperformed the baseline in accuracy (ACC) across various configurations:

  • Large-Scale Linking: In tests with 10,000 utterances, DTVAE reached 55.6% ACC, surpassing the baseline's 52.2%.
  • Fixed Category Clustering: When the number of speakers is known, the improvements are even more stark. For a 3-speaker scenario, DTVAE achieved 93.3% ACC, compared to 86.6% for the baseline.

Conclusion and Future Work

DTVAE represents a significant step toward robust, scalable speaker clustering. By leveraging non-linear neural mappings and discrete latent spaces, it overcomes the limitations of traditional linear models and the $O(N^2)$ complexity of standard AHC. While accuracy still faces challenges as speaker counts increase, future work aims to evolve DTVAE into a fully end-to-end system, potentially eliminating the need for secondary clustering steps entirely.

Summary updated

Abstract

Recently, the speaker clustering model based on aggregation hierarchy cluster (AHC) is a common method to solve two main problems: no preset category number clustering and fix category number clustering. In general, model takes features like i-vectors as input of probability and linear discriminant analysis model (PLDA) aims to form the distance matric in long voice application scenario, and then clustering results are obtained through the clustering model. However, traditional speaker clustering method based on AHC has the shortcomings of long-time running and remains sensitive to environment noise. In this paper, we propose a novel speaker clustering method based on Mutual Information (MI) and a non-linear model with discrete variable, which under the enlightenment of Tied Variational Autoencoder (TVAE), to enhance the robustness against noise. The proposed method named Discrete Tied Variational Autoencoder (DTVAE) which shortens the elapsed time substantially. With experience results, it outperforms the general model and yields a relative Accuracy (ACC) improvement and significant time reduction.
BibTeX
@inproceedings{feng2020robust,
  author = {Chen Feng and Jianzong Wang and Tongxu Li and Junqing Peng and Jing Xiao},
  title = {{A Robust Speaker Clustering Method Based on Discrete Tied Variational Autoencoder}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=gTl8FZcOvhkv},
}
Figure