Exploiting Cross-Lingual Speaker and Phonetic Diversity for Unsupervised Subword Modeling

Tan Lee 李丹, Siyuan Feng, Tan LeeIEEE/ACM Trans. ASLP 2019 · DOI

TL;DR

This paper tackles the challenge of building speech recognition models for languages with very little or no transcribed data. They propose a novel multi-task deep learning framework that leverages out-of-domain languages and speaker adaptation techniques to learn robust subword representations, achieving competitive performance on the ZeroSpeech 2017 challenge.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems have made significant strides, but their success heavily relies on vast amounts of transcribed speech data. This is a major bottleneck for low-resource languages, where such data is scarce or non-existent. The problem of "unsupervised subword modeling" or "zero-resource speech modeling" aims to build acoustic models for these languages without any transcribed training data. The core challenge lies in learning meaningful representations of speech units (subwords or phonemes) that are robust to variations in speakers, accents, and recording conditions, all while relying solely on unlabeled audio.

Methods

The authors propose a framework that combines several key techniques:

  • Speaker-Adapted Feature Extraction: They use out-of-domain (resource-rich) languages to train ASR systems. These systems are then used to perform speaker adaptation on the target language's unlabeled speech, generating features like fMLLRs that are more robust to speaker variations.
  • Unsupervised Acoustic Modeling (DPGMM-HMM): To generate initial frame-level labels for the unlabeled target speech, they employ Dirichlet Process Gaussian Mixture Models (DPGMM) for clustering speech frames. To address the issue of DPGMM producing over-fragmented clusters and to incorporate temporal dependencies, they further refine this by training a Hidden Markov Model (HMM) on these DPGMM clusters, creating "DPGMM-HMM" labels.
  • Label Filtering: A novel algorithm is introduced to filter out infrequent or potentially erroneous labels generated by DPGMM clustering, aiming to improve the quality of training data for subsequent models.
  • Multi-Task Deep Neural Network (MTL-DNN) Learning: This is the central component. A DNN is trained with multiple tasks simultaneously. These tasks include predicting the DPGMM-HMM alignments for the target languages and predicting senone labels from out-of-domain ASR systems. By sharing a low-dimension bottleneck (BNF) layer across these tasks, the DNN learns representations that are generalizable and robust. Different types of BNFs are generated based on the tasks included:
  • MUBNF (Multilingual Unsupervised BNF): Trained using only target language DPGMM-HMM labels.
  • OSBNF (Out-of-domain Supervised BNF): Trained using senone labels from out-of-domain ASR systems.
  • LI-BNF (Language-Independent BNF): Trained using both target language DPGMM-HMM labels and out-of-domain ASR senone labels.
  • Transfer Learning BNF (TLBNF): Features extracted from a pre-trained ASR system of a resource-rich language are also used.

These different BNF representations can be concatenated to further improve performance.

Datasets

  • Training: The experiments utilize development data from the ZeroSpeech 2017 Challenge. This includes untranscribed speech for three target languages: English, French, and Mandarin.
  • Out-of-Domain Data: For training out-of-domain ASR systems, the paper mentions using the CUSENT database for Cantonese, and other phone recognizers trained on SpeechDat-E databases for Czech, Hungarian, and Russian.
  • Evaluation: The evaluation is performed on the test sets of the ZeroSpeech 2017 Challenge, using the ABX subword discriminability metric.

Key Insights and Ideas

  • Leveraging Out-of-Domain Data is Crucial: The study strongly demonstrates that using transcribed data from resource-rich languages (even if linguistically distant) significantly benefits the learning of robust features for low-resource languages.
  • Complementarity of Labeling Strategies: Combining DPGMM-based clustering (capturing acoustic properties) with out-of-domain ASR decoding (capturing phonetic knowledge) in a multi-task learning framework leads to more effective and language-independent representations.
  • Speaker Adaptation is Key: Applying speaker adaptation techniques, even using out-of-domain data, is vital for creating features that are invariant to speaker variations.
  • Temporal Context Matters: Incorporating temporal dependencies (via DPGMM-HMM) in label generation yields better results than frame-level clustering alone.
  • Concatenation of Features Boosts Performance: Combining different types of learned BNF representations (e.g., unsupervised, supervised, transfer learning) leads to the best overall performance.

Key Results

  • The proposed system achieved an across-speaker triphone minimal-pair ABX error rate of 9.7%, which was competitive with the best systems reported for the ZeroSpeech 2017 Challenge.
  • The use of out-of-domain fMLLR features consistently outperformed the generic MFCC baseline.
  • Multilingual BNFs (MUBNF) learned via MTL-DNN showed significant improvement over the input features (MFCC and fMLLR).
  • Adding out-of-domain ASR senone labels (OSBNF and LI-BNF) further reduced the ABX error rates.
  • Concatenating HMM(P)-MUBNF, OSBNF1, and TLBNF achieved the best results, demonstrating the benefit of combining different feature learning strategies.
  • The proposed label filtering algorithm showed a slight improvement when removing a small percentage (10-30%) of labels, indicating that some labels are indeed unreliable.

Limitations and Future Work

  • Language Closeness and Data Amount: The paper notes that the closeness between target and out-of-domain languages, as well as the amount of available training data for individual target languages, can significantly impact the quality of learned features. For instance, OSBNF2 performed worse than OSBNF1 on Mandarin, suggesting that highly mismatched languages might not always be beneficial.
  • Label Filtering Simplicity: The proposed label filtering is simple, relying only on label counts. The authors suggest that more sophisticated methods might be explored, potentially considering the meaning of infrequent subword units.
  • DNN Model Structures: While MLP performed best in their experiments, they acknowledge that LSTM and BLSTM are generally better at capturing temporal dependencies. They suggest that with more data, these recurrent structures might yield better results.
  • Future Work: The authors imply that their methods are generalizable to other language combinations beyond those in the ZeroSpeech 2017 challenge. Further exploration into optimal combinations of features and more advanced label filtering techniques could be promising.

Summary updated

BibTeX
@article{2019exploiting,
  author = {Tan Lee 李丹 and Siyuan Feng and Tan Lee},
  title = {{Exploiting Cross-Lingual Speaker and Phonetic Diversity for Unsupervised Subword Modeling}},
  journal = {IEEE/ACM Trans. ASLP},
  year = {2019},
  doi = {10.1109/taslp.2019.2937953},
}
Figure