SparseSpeech: Unsupervised Acoustic Unit Discovery with Memory-Augmented Sequence Autoencoders

Benjamin Milde, Chris BiemannInterspeech 2019

TL;DR

This paper introduces SparseSpeech, a novel unsupervised method for discovering acoustic units in speech without needing transcriptions. It uses a memory-augmented sequence autoencoder that learns to represent speech sparsely, leading to more compact and discriminative acoustic units, outperforming traditional methods like MFCCs.

What Problem Does This Paper Address?

The field of speech processing often relies on vast amounts of transcribed speech data, which is expensive and time-consuming to create. This is particularly challenging for low-resource languages. Unsupervised acoustic unit discovery aims to automatically identify the fundamental building blocks of speech (like phonemes or sub-word units) directly from raw audio, enabling tasks like voice search, lexical discovery, and speech recognition in languages with limited resources. A key challenge is handling the inherent variability in speech due to different speakers, environments, and recording conditions, which makes direct comparisons difficult. Existing methods often struggle to produce representations that are both compact and discriminative enough for downstream tasks.

Methods

The core of SparseSpeech is a sequence autoencoder architecture built upon bidirectional LSTMs. What makes it novel is the integration of a memory component that acts as a sparsity-inducing bottleneck. This memory component is a differentiable embedding memory bank that is addressed by the encoder.

Here's a breakdown of the key techniques:

  • Sequence Autoencoder: The model learns to encode an input speech sequence into a compressed representation and then decode it back to reconstruct the original input.
  • Memory-Augmented Bottleneck: Instead of a standard dense bottleneck, SparseSpeech uses a memory module. The encoder's output queries this memory, which contains learned "value" embeddings. This allows for a more structured and potentially symbolic representation.
  • Sparsity Constraint: A crucial innovation is the use of an L∞ regularization on the softmax output of the memory addressing network. This encourages the model to activate only a few memory locations for each input frame, leading to sparse, "one-hot-like" representations. The degree of sparsity can be controlled by a hyperparameter.
  • Sequence Dropout: To encourage the model to learn temporal structure, random "sequence dropout" is applied, where parts of the encoded sequence are omitted during training. This forces the decoder to infer missing information and rely on its internal state.
  • Diversity Constraint: To prevent the model from collapsing into a trivial solution (e.g., mapping everything to a single unit), a Kullback-Leibler (KL) divergence-based diversity constraint is introduced.
  • Iterative Training Procedure: Training the full memory-augmented autoencoder from scratch proved difficult. The authors propose an iterative approach: first, train the autoencoder without the sparsity layer, then use clustering (like k-means or DPGMM) on the bottleneck features to initialize the memory bank, and finally, train the full network with the sparsity and diversity constraints.

Datasets

  • Training: The model was trained on the 360-hour clean subset of the Librispeech corpus for English read speech.
  • Evaluation:
    • ABX Discriminability: The primary evaluation metric is the ABX task, which measures how well discovered units can discriminate between similar speech sounds. This was performed on the Librispeech corpus.
  • ZeroSpeech 2017 Test Set: The model was also evaluated on the English test set from the ZeroSpeech 2017 challenge.

Key Insights and Ideas

  • Memory as a Sparsity Inducer: The memory component, when combined with sparsity constraints, is highly effective at forcing the autoencoder to learn compact, symbolic-like representations, moving beyond dense, continuous embeddings.
  • Controlled Sparsity: The ability to control the sparsity level via a loss function parameter is a powerful feature, allowing for a trade-off between representation compactness and reconstruction fidelity.
  • Iterative Training is Key: The proposed iterative training strategy is essential for successfully training the complex memory-augmented architecture, overcoming initial training instabilities.
  • Context Matters: Incorporating an utterance-level context vector (e.g., the mean of encoder states) helps the decoder disambiguate acoustic units by providing information about the overall "style" of the utterance.
  • Diversity Prevents Collapse: The diversity constraint is crucial to ensure that the learned units are meaningful and cover a range of acoustic variations, rather than collapsing into a few dominant ones.

Key Results

  • Improved ABX Scores: SparseSpeech achieved competitive or superior ABX discriminability scores compared to baseline methods like MFCC and PLP, especially in across-speaker evaluations.
  • Compact Representations: The sparsity constraint led to representations that were "mostly one-hot encoded" and "symbolic-like," significantly more compact than dense embeddings.
  • Scalability: The model was trained on 360 hours of speech in two days on a single GPU, demonstrating good scalability.
  • Comparison to PLP-DPGMM: While PLP-DPGMM is a strong baseline, SparseSpeech achieved better ABX discriminability when restricted to sparse, symbolic outputs, and was easier to scale to larger datasets.
  • Effectiveness of Training Strategy: The iterative training procedure significantly improved performance compared to direct end-to-end training.

Limitations and Future Work

  • Initialization Sensitivity: While the iterative training helps, the initialization of the memory component (e.g., using k-means or DPGMM) can still influence performance.
  • Hyperparameter Tuning: The sparsity weight and the number of memory units (n) are hyperparameters that need careful tuning. The paper suggests that n could potentially be inferred from data using density-based clustering.
  • Future Work: The authors suggest exploring different clustering algorithms for initialization, inferring the number of units from data, and further investigating the impact of different memory architectures and sparsity regularization techniques. They also highlight the potential for this approach in low-resource scenarios.

Summary updated

BibTeX
@inproceedings{milde2019sparsespeech,
  author = {Benjamin Milde and Chris Biemann},
  title = {{SparseSpeech: Unsupervised Acoustic Unit Discovery with Memory-Augmented Sequence Autoencoders}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure