Abstract
In this paper, a novel neural network layer is proposed to combine frame-level into utterance-level representations for speaker modeling. We followed the assumption that the frame-level outputs of the speaker embedding (a.k.a x-vector) encoder are multi-modal. Therefore, we modeled the frame-level information as a mixture of factor analyzers with latent variable (utterance embedding) tied across frames and mixture components, in a similar way as in the i-vector approach. We denote this layer as Tied Mixture of Factor Analyzers (TMFA) layer. The optimal value of the embedding is obtained by minimizing the reconstruction error of the frame-level representations given the embedding and the TMFA model parameters. However, the TMFA layer parameters (factor loading matrices, means and precisions) were trained with cross-entropy loss as the rest of parameters of the network. We experimented on the Speaker Recognition Evaluation 2016 Cantonese as well as in the Speaker in the Wild datasets. The proposed pooling layer improved w.r.t. mean plus standard deviation pooling — standard in x-vector approach — in most of the conditions evaluated; and obtained competitive performance w.r.t. the recently proposed learnable dictionary encoding pooling method, which also assumes multi-modal frame-level representations.
Introduction
In modern speaker recognition, the transition from classical i-vectors to deep neural network-based "x-vectors" has significantly improved performance. A critical step in creating these speaker embeddings is "pooling," where thousands of short-duration frame-level representations are condensed into a single utterance-level vector.
The standard approach, statistical pooling (SP), calculates the mean and standard deviation of these frames. However, this method assumes a simple, unimodal distribution of speech data. In reality, speech is inherently multi-modal, containing diverse phonetic content, background noise, and emotional states. While methods like Learnable Dictionary Encoding (LDE) have been introduced to address this multi-modality, they do not fully leverage the mathematical robustness of classical factor analysis. This paper introduces the Tied Mixture of Factor Analyzers (TMFA) layer to bridge this gap by bringing i-vector logic into the neural embedding architecture.
The Tied Mixture of Factor Analyzers (TMFA) Layer
The TMFA layer treats pooling as a reconstruction problem rather than a simple statistical aggregation. It assumes that frame-level representations belong to different clusters (components), similar to a Gaussian Mixture Model (GMM).
Key aspects of the architecture include:
- Latent Space Modeling: Each frame is modeled as a combination of a cluster mean and a low-rank "total variability" matrix multiplied by a latent speaker embedding ($v$). This embedding is "tied" or kept constant across all frames in an utterance.
- Closed-Form Solution: Instead of simple averaging, the TMFA layer finds the embedding $v$ that minimizes the reconstruction error across all frames and clusters. This leads to a familiar closed-form solution ($v = A^{-1}b$) used in i-vector research.
- Differentiability: To allow for end-to-end training, the authors derived gradients for the matrix inversion and utilized Cholesky decomposition to ensure computational efficiency during forward and backward passes.
Experimental Setup
The models were trained using a ResNet-34 backbone on a massive combined dataset including Switchboard (Phases 1, 2, and 3, plus Cellular) and NIST SRE data from 2004–2010, totaling approximately 7,000 speakers. Data was augmented with babble, music, noise, and reverberation to improve robustness.
Evaluation was conducted on two benchmarks:
- SRE16 Cantonese: Focused on conversational telephone speech.
- Speakers in the Wild (SITW): Real-world audio with varying lengths (6–240 seconds).
Results and Discussion
The TMFA layer demonstrated superior performance in classification accuracy during training. As shown in the table below, the TMFA layer consistently outperformed both Statistical Pooling and LDE across different audio chunk lengths.
Specifically, for 5-8 second chunks, TMFA achieved a validation accuracy of 94.12%, compared to 91.17% for LDE and 87.58% for SP. This confirms that accounting for multi-modality is crucial for high-fidelity speaker modeling. Furthermore, the performance gains observed with longer audio chunks suggest that TMFA effectively leverages the additional phonetic and acoustic variety present in longer utterances to better observe mixture components.
On the SITW evaluation set, TMFA achieved an Equal Error Rate (EER) of 5.74%, significantly outperforming the standard ResNet34+SP (6.13%), though it trailed slightly behind LDE (5.29%) in this specific metric.
Conclusion
The TMFA layer successfully applies the principles of factor analysis to the hidden layers of a deep neural network. While it achieves state-of-the-art validation accuracy, the slight disconnect between training accuracy and final EER suggests that future work should focus on aligning training objectives more closely with the verification task. Additionally, stabilizing the matrix inversion process to prevent rank deficiency remains a priority for further refining this mathematically rigorous pooling approach.
Summary updated