A comparison of neural network methods for unsupervised representation learning on the zero resource speech challenge

Daniel Renshaw, Herman Kamper, Aren Jansen, Sharon GoldwaterInterspeech 2015

TL;DR

This paper explores unsupervised neural network methods for learning speech representations, crucial for languages lacking extensive transcribed data. They demonstrate that a "correspondence autoencoder" (cAE) significantly outperforms standard autoencoders and even denoising autoencoders, especially when optimized for a specific task and language, and can be further enhanced by combining with supervised features from a high-resource language.

What Problem Does This Paper Address?

Traditional speech recognition systems rely heavily on massive amounts of transcribed speech data. This makes them ineffective for the vast majority of the world's languages, which are "zero-resource" due to a lack of such data. The challenge is to develop methods that can learn useful speech representations without requiring transcriptions, mimicking how human infants learn language. This paper specifically tackles the problem of learning frame-level representations that can discriminate between subword units, a fundamental step towards building speech technologies for low-resource languages.

Methods

The authors investigate several unsupervised neural network architectures for learning speech representations:

  • Autoencoder (AE): A standard AE learns to reconstruct its input. The paper uses both shallow and deep AE architectures, noting that deep AEs generally perform better.
  • Denoising Autoencoder (dAE): This variant is trained to reconstruct clean inputs from noisy versions. The paper suggests dAEs offer better regularization than standard AEs, preventing degenerate solutions.
  • Correspondence Autoencoder (cAE): This is the core novel contribution. A cAE is a weakly supervised variant of an AE. Instead of reconstructing its own input, it's trained to reconstruct a similar input. This "similarity" is derived from an unsupervised term discovery (UTD) system, which finds pairs of speech segments likely to contain the same word. The cAE uses these paired frames to learn representations that better approximate the downstream task's objective. The authors explore two cAE architectures: one with a shallow decoder (similar to previous work) and one with a deep, non-linear decoder, which allows the top encoder layer to be directly used as the representation.
  • Bottleneck Features (BNFs): As a comparison, the authors also extract features from the bottleneck layer of a supervised deep neural network (DNN) trained on a large English corpus. These BNFs are then used as input to their unsupervised models or combined with them.

Datasets

  • Buckeye Corpus: A 5-hour portion of this English corpus was used. It was part of the Zero Resource Speech Challenge (ZRSC).
  • NCHLT Xitsonga Speech Corpus: A 2.5-hour portion of this Xitsonga (a Southern African Bantu language) corpus was used, also for the ZRSC.
  • Switchboard Corpus: Used for pre-training the DNN that generates BNFs.

The ZRSC defines two tasks, and this paper focuses on Track 1: Subword Modeling, evaluated using the minimal triphone pair ABX discrimination task. This task measures how well frame representations can distinguish between similar speech units.

Key Insights and Ideas

  • cAEs are superior for unsupervised representation learning: The paper strongly supports the idea that using "correspondence pairs" (even if weakly supervised) provides a significant advantage over standard autoencoders or denoising autoencoders for learning discriminative speech features.
  • Task and language optimization matters: Optimizing the cAE architecture and training regime for the specific evaluation task (ABX) and language (Buckeye English) yielded better results than using a generic architecture optimized on a different task (Switchboard/same-different).
  • Cross-lingual transfer is promising: A supervised DNN trained on English can produce useful BNFs for Xitsonga, demonstrating a degree of cross-lingual generalization.
  • Combining supervised and unsupervised methods yields the best results: Applying the unsupervised cAE to the cross-lingual BNFs resulted in the best performance on Xitsonga, suggesting a powerful synergy between supervised learning in high-resource settings and unsupervised adaptation to low-resource languages.
  • Deep nonlinear decoders are beneficial: Using a deep, nonlinear decoder in the cAE allows the top encoder layer to be directly used as the representation, simplifying the process and avoiding the need for explicit layer selection.

Key Results

  • The "Optimal" cAE, optimized on Buckeye English for the ABX task, achieved a 35% relative error rate reduction compared to MFCCs on the Xitsonga zero-resource task.
  • Combining the "Optimal" cAE with English-trained DNN BNFs for Xitsonga resulted in the best performance, achieving an overall relative error rate reduction of 39% over MFCCs.
  • The cAE consistently outperformed denoising autoencoders and standard autoencoders across various configurations.
  • The "Original" cAE (optimized on Switchboard English) still provided significant improvements on Xitsonga (26% relative error reduction), demonstrating good cross-language transferability even without task-specific optimization.

Limitations and Future Work

  • Correspondence data availability: The performance of the cAE is dependent on the quality and quantity of correspondence pairs generated by the UTD system. If UTD fails to find good pairs, the cAE's effectiveness is limited.
  • UTD system limitations: The paper notes that while the cAE is a version of a dAE where "true sources of extrinsic variability" are used as noise, this is limited by what the UTD system can discover.
  • Future work: The authors suggest exploring different types and levels of noise for dAEs, which could be beneficial when correspondence pairs are unavailable. They also highlight the promising direction of combining supervised training in high-resource languages with unsupervised language adaptation for zero-resource settings. Further optimization of cAE architectures for specific zero-resource languages is also a potential avenue.

Summary updated

BibTeX
@inproceedings{renshaw2015comparison,
  author = {Daniel Renshaw and Herman Kamper and Aren Jansen and Sharon Goldwater},
  title = {{A comparison of neural network methods for unsupervised representation learning on the zero resource speech challenge}},
  booktitle = {Interspeech},
  year = {2015},
}
Figure