Sampling strategies in Siamese Networks for unsupervised speech representation learning

Rachid Riad, Corentin Dancette, Julien Karadayi, Neil Zeghidour, Thomas Schatz, Emmanuel DupouxarXiv 2018 · arXiv

Introduction

In the field of unsupervised speech processing, the goal is to discover the underlying phonetic structure of a language without the aid of transcriptions or dictionaries. Siamese networks have emerged as a powerful tool for this task, learning embeddings by comparing pairs of audio segments to determine if they represent the same or different words. While significant research has focused on neural architectures and loss functions, this study by Riad et al. (presented at Interspeech 2018) highlights a critical but often overlooked factor: the sampling strategy. The authors demonstrate that how training pairs are selected is as vital as the model architecture itself.

The Challenge of Zipfian Distributions

Natural language is characterized by highly unbalanced word frequency distributions, a phenomenon known as Zipf’s Law. A small number of high-frequency words (e.g., "the," "and," "is") dominate any given corpus, while the vast majority of words appear only a few times.

Standard random sampling techniques naturally follow this distribution. Consequently, a Siamese network trained on random pairs spends a disproportionate amount of time processing a limited set of phonetic patterns. The authors argue that this lack of phonetic diversity prevents the model from learning robust, invariant representations that generalize across the entire language. To combat this, they propose "flattening" the distribution to ensure rare words receive adequate attention during training.

Sampling Strategies and Compression Functions

The core of the methodology involves the ABnet architecture, which processes filterbank features into 100-dimensional embeddings. The researchers introduced a sampling compression function, $\phi(n)$, to adjust the probability of selecting a word type based on its frequency $n$ in the dataset. They tested several variations:

  • Linear ($\phi(n) = n$): The baseline, following the natural Zipfian distribution.
  • Power Laws ($\phi(n) = \sqrt{n}$ or $\sqrt[3]{n}$): Moderate flattening of the distribution.
  • Logarithmic ($\phi(n) = \log(1+n)$): Stronger compression.
  • Uniform ($\phi(n) = 1$): Complete flattening, where every word type has an equal probability of being sampled regardless of its frequency.

Key Insights: Quality Over Quantity

The experimental results revealed that the most aggressive compression—the Uniform strategy—consistently yielded the best performance. By treating rare words with the same importance as common ones, the network was forced to capture a broader range of phonetic features, leading to more discriminative embeddings.

Impact of sampling compression functions on ABX error rates across different data scales.

One of the most striking findings was the "More Data Trap." As shown in Figure 2, using the Uniform sampling strategy on just 1% of the Buckeye Corpus (approximately 3 minutes of audio) often outperformed the standard Linear strategy applied to the full 100% dataset (nearly 5 hours). This suggests that in unsupervised learning, the diversity and quality of the training pairs are more significant than the sheer volume of data.

Furthermore, the authors explored the impact of speaker and word-type ratios. Contrary to intuition, training on pairs from the same speaker ($P^s_- = 0$) proved more effective than across-speaker pairs. This is likely because the Dynamic Time Warping (DTW) used for alignment is more accurate within a single speaker's voice, providing cleaner supervision. Additionally, they found that a slight bias toward negative samples (different word types, $P^w_- \approx 0.7$) optimized the model's discriminative power.

Comparative Performance

The optimized ABnet, utilizing uniform sampling and tuned ratios, achieved state-of-the-art results on the Zero Resource Speech Challenge 2015 tasks. It significantly outperformed previous Siamese network iterations and Correspondence Auto-Encoders (CAE).

Performance comparison of the optimized ABnet against baseline and state-of-the-art unsupervised models.

As detailed in Figure 5, the model achieved an ABX error rate of 10.4% for within-speaker English tasks. Crucially, the strategies developed on English transferred successfully to Xitsonga, a language from a completely different family (Bantu). This generalization suggests that these sampling principles are universal across human languages.

Conclusion

This research underscores that the effectiveness of Siamese networks for speech is heavily dependent on the sampling distribution. By neutralizing the effects of Zipf’s Law through uniform sampling, researchers can achieve superior phonetic representations even with minimal data. While some generative models like DPGMMs still hold advantages in specific categories, the improvements in ABnet provide a clear path forward for discriminative unsupervised learning. Future work will focus on integrating these improved representations back into Spoken Term Discovery systems to create a virtuous cycle of better data and better models.

Summary updated

Abstract

Recent studies have investigated siamese network architectures for learning invariant speech representations using same-different side information at the word level. Here we investigate systematically an often ignored component of siamese networks: the sampling procedure (how pairs of same vs. different tokens are selected). We show that sampling strategies taking into account Zipf's Law, the distribution of speakers and the proportions of same and different pairs of words significantly impact the performance of the network. In particular, we show that word frequency compression improves learning across a large range of variations in number of training pairs. This effect does not apply to the same extent to the fully unsupervised setting, where the pairs of same-different words are obtained by spoken term discovery. We apply these results to pairs of words discovered using an unsupervised algorithm and show an improvement on state-of-the-art in unsupervised representation learning using siamese networks.
BibTeX
@misc{riad2018sampling,
  author = {Rachid Riad and Corentin Dancette and Julien Karadayi and Neil Zeghidour and Thomas Schatz and Emmanuel Dupoux},
  title = {{Sampling strategies in Siamese Networks for unsupervised speech representation learning}},
  year = {2018},
  eprint = {1804.11297},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1804.11297},
}
Figure