TL;DR
This paper introduces Neural Hidden Markov Models (NHMMs) to capture temporal dependencies between discrete speech representations, improving phonetic information accessibility and segmentation compared to models that assume frame independence. By extending existing self-supervised learning frameworks, NHMMs offer a more structured approach to learning speech representations that better aligns with the sequential nature of human speech.
What Problem Does This Paper Address?
Self-supervised learning for speech representation has seen great success using discrete latent variable models. However, a common assumption in these models is that consecutive frames are independent. This assumption is at odds with the inherent sequential and segmental nature of speech, where phonemes and their transitions carry crucial information. The paper aims to bridge this gap by explicitly modeling dependencies between discrete latent variables, hypothesizing that this will lead to richer and more useful speech representations for downstream tasks like phonetic recognition and segmentation.
Methods
The core contribution is the development of Neural Hidden Markov Models (NHMMs). NHMMs build upon existing neural network architectures used for self-supervised learning (like VQ-APC) but introduce a Markovian dependency structure between the discrete latent variables. This means the probability of a current latent state depends on the previous latent state, rather than being independent.
Key methodological aspects include:
- Probabilistic Graphical Models: The dependencies are framed within the language of probabilistic graphical models, specifically using the structure of Hidden Markov Models (HMMs).
- Neural Parametrization: The transition and emission probabilities of the HMM are parametrized using neural networks, allowing for flexible and data-driven learning. Crucially, the transition distribution is designed to depend on past acoustic frames, not just the previous latent state.
- Low Frame-Rate HMMs (Hops): To explicitly model longer-range dependencies and mimic the slower rate of phonetic events compared to acoustic frames, the authors introduce "hops." This involves using multiple parallel HMM chains, each modeling latent variables at intervals of H frames, effectively creating a low frame-rate HMM.
- Training: The models are trained end-to-end by maximizing the log-likelihood using a custom implementation of the forward-backward algorithm for efficient gradient computation.
The paper also notes that their NHMM framework can subsume VQ-APC by removing the Markovian dependency, allowing for a direct comparison of the impact of these dependencies.
Datasets
- LibriSpeech (train-clean-360): Used for pre-training the models.
- Wall Street Journal (WSJ): Used for phone classification and phone cluster purity evaluation.
- TIMIT: Used for phone segmentation evaluation.
The authors mention following similar settings to prior work for these datasets, with specific splits for training and evaluation (e.g., 90% of si284 for WSJ training).
Key Insights and Ideas
- Dependencies Matter: Explicitly modeling temporal dependencies between discrete latent variables significantly improves the quality of learned speech representations, particularly for phonetic tasks.
- Aligning with Speech Structure: The sequential nature of NHMMs better reflects the segmental structure of speech compared to frame-independent models.
- "Hops" for Phonetic Timescales: Introducing hops in NHMMs, with hop lengths around the average phone duration, leads to improved performance, suggesting that modeling at a slower, phonetic timescale is beneficial.
- VQ-APC as a Special Case: The ability to revert the NHMM to a VQ-APC model by removing dependencies provides a clean way to isolate and demonstrate the benefits of the proposed Markovian assumptions.
Key Results
- Improved Phone Classification: Neural HMMs with appropriate hop lengths consistently outperformed VQ-APC (both with Gumbel approximation and marginalization) on phone classification tasks. For instance, with a codebook size of 100 and a hop length of 7, NHMMs achieved a PER of 19.7%, compared to VQ-APC's 20.3%.
- Higher Cluster Purity: NHMMs demonstrated better normalized mutual information (NMI) compared to VQ-APC, indicating higher purity of learned phonetic clusters.
- Enhanced Phone Segmentation: NHMMs showed significant improvements in phone segmentation accuracy (F1-score) over VQ-APC. The paper suggests that fewer codes lead to fewer transitions, and more codes reveal finer phonetic information, consistent with better PER and NMI.
- Qualitative Code Analysis: The learned discrete codes in NHMMs showed a correlation with broad phonetic categories (vowels, closures, fricatives, nasals, silence), even when trained on a general dataset and evaluated on TIMIT.
Limitations and Future Work
- Custom Implementation: The authors developed a custom C++ implementation of the forward-backward algorithm to speed up computation, as standard automatic differentiation in PyTorch was found to be slow. This custom implementation is noted as being publicly available.
- Parametrization Choices: The paper focuses on a specific parametrization for the transition distribution. They acknowledge that other, more complex parametrizations involving more layers could be explored in future work.
- Segmentation Nuances: While NHMMs improve phone segmentation, the authors note it's "less clear whether segmentation can be achieved by simply using the learned HMMs" directly, suggesting further investigation into how the learned representations are best utilized for this task.
- Hyperparameter Tuning: The optimal hop length was found to be around the average phone duration, but the paper implies that further study on the effect of hops on representation accessibility is warranted.
Summary updated