Abstract
In many speech processing applications, the objective is to model different modes of variability to obtain robust speech features. In this paper, we learn speech representations in a multiview paradigm by constraining the views to known modes of variability such as speakers or spoken words. We use deep multiset canonical correlation (dMCCA) because it can model more than two views in parallel to learn a shared subspace across them. In order to model thousands of views (e.g., speakers), we demonstrate that stochastically sampling a small number of views generalizes dMCCA to the larger set of views. To evaluate our approach, we study two different aspects of the Speech Commands Dataset: variability among the speakers and speech commands. We show that, by treating observations from one mode of variability as multiple parallel views, we can learn representations that are discriminative to the other mode. We first consider different speakers as views of the same word to learn their shared subspace to represent an utterance. We then constrain the different words spoken by the same person as multiple views to learn speaker representations. Using classification and unsupervised clustering, we evaluate the efficacy of multiview representations to identify speech commands and speakers.
Introduction
Speech signals are inherently complex, containing multiple overlapping modes of variability. A single utterance like "TV On" encodes not only the linguistic command but also the speaker's unique vocal characteristics, room acoustics, and ambient noise. Traditionally, speech processing has relied on "total variability modeling" (e.g., i-vectors) to capture these factors, often requiring complex post-processing to isolate the desired signal.
This work proposes a multiview learning paradigm to explicitly model these modes of variability. By treating different instances of the same underlying event—such as different speakers saying the same word—as multiple "views," we can learn a shared subspace that captures the common information (the command) while discarding view-specific noise (the speaker identity).
Methodology
The core of this approach is Deep Multiset Canonical Correlation Analysis (dMCCA). In this framework, multiple neural network branches (Convolutional Neural Networks) act as view-specific encoders that project input signals into a common latent subspace.
Generalized dMCCA and Stochastic Sampling
A significant challenge in multiview learning for speech is the scale of views. For instance, a dataset might contain thousands of speakers. Standard dMCCA, which requires a dedicated branch for every view, is computationally infeasible at this scale. To address this, we introduced Generalized dMCCA. During training, we stochastically sample a small number of views ($m$, typically between 3 and 8) from the total pool. This allows the model to learn a generalized shared subspace that scales to thousands of views without a proportional increase in parameters. At inference time, the model is view-agnostic; a single branch can generate robust embeddings for unseen audio.
Experimental Evaluation
We evaluated our approach using the Google Speech Commands Dataset, which includes over 1,800 speakers and 30 short commands. We defined two primary tasks:
- Command Identification (Command-ID): Using different speakers as views to learn word-discriminative features.
- Speaker Identification (Speaker-ID): Using different words spoken by the same individual as views to learn speaker-discriminative features.
Results
Our dMCCA approach significantly outperformed traditional industry standards like i-vectors and x-vectors, particularly in scenarios with short audio segments (~1 second). For Command-ID, dMCCA achieved a Macro F1 score of 0.94, compared to 0.89 for x-vectors. The gap was even wider in Speaker-ID, where dMCCA hit a purity score of 0.92, while x-vectors trailed at 0.48.
Discussion: Phonetic Proximity
Qualitative analysis via t-SNE visualizations reveals that the learned command representations capture deep phonetic relationships. The model does not merely cluster identical words; it places phonetically similar words in close proximity within the manifold. For example, "down" and "dog" are clustered near each other, as are "on," "off," and "up."
Conclusion
By leveraging multiview shared subspace learning, we can effectively decouple speaker identity from linguistic content. The proposed stochastic sampling method makes dMCCA viable for large-scale datasets, providing a robust alternative to traditional variability modeling. Future research will explore treating environmental noise and channel distortions as additional views to further enhance the robustness of speech representations.
Summary updated