Introduction
In the field of unsupervised speech processing, the goal is to discover the underlying phonetic structure of a language without the aid of transcriptions or dictionaries. Siamese networks have emerged as a powerful tool for this task, learning embeddings by comparing pairs of audio segments to determine if they represent the same or different words. While significant research has focused on neural architectures and loss functions, this study by Riad et al. (presented at Interspeech 2018) highlights a critical but often overlooked factor: the sampling strategy. The authors demonstrate that how training pairs are selected is as vital as the model architecture itself.
The Challenge of Zipfian Distributions
Natural language is characterized by highly unbalanced word frequency distributions, a phenomenon known as Zipf’s Law. A small number of high-frequency words (e.g., "the," "and," "is") dominate any given corpus, while the vast majority of words appear only a few times.
Standard random sampling techniques naturally follow this distribution. Consequently, a Siamese network trained on random pairs spends a disproportionate amount of time processing a limited set of phonetic patterns. The authors argue that this lack of phonetic diversity prevents the model from learning robust, invariant representations that generalize across the entire language. To combat this, they propose "flattening" the distribution to ensure rare words receive adequate attention during training.
Sampling Strategies and Compression Functions
The core of the methodology involves the ABnet architecture, which processes filterbank features into 100-dimensional embeddings. The researchers introduced a sampling compression function, $\phi(n)$, to adjust the probability of selecting a word type based on its frequency $n$ in the dataset. They tested several variations:
- Linear ($\phi(n) = n$): The baseline, following the natural Zipfian distribution.
- Power Laws ($\phi(n) = \sqrt{n}$ or $\sqrt[3]{n}$): Moderate flattening of the distribution.
- Logarithmic ($\phi(n) = \log(1+n)$): Stronger compression.
- Uniform ($\phi(n) = 1$): Complete flattening, where every word type has an equal probability of being sampled regardless of its frequency.
Key Insights: Quality Over Quantity
The experimental results revealed that the most aggressive compression—the Uniform strategy—consistently yielded the best performance. By treating rare words with the same importance as common ones, the network was forced to capture a broader range of phonetic features, leading to more discriminative embeddings.
One of the most striking findings was the "More Data Trap." As shown in Figure 2, using the Uniform sampling strategy on just 1% of the Buckeye Corpus (approximately 3 minutes of audio) often outperformed the standard Linear strategy applied to the full 100% dataset (nearly 5 hours). This suggests that in unsupervised learning, the diversity and quality of the training pairs are more significant than the sheer volume of data.
Furthermore, the authors explored the impact of speaker and word-type ratios. Contrary to intuition, training on pairs from the same speaker ($P^s_- = 0$) proved more effective than across-speaker pairs. This is likely because the Dynamic Time Warping (DTW) used for alignment is more accurate within a single speaker's voice, providing cleaner supervision. Additionally, they found that a slight bias toward negative samples (different word types, $P^w_- \approx 0.7$) optimized the model's discriminative power.
Comparative Performance
The optimized ABnet, utilizing uniform sampling and tuned ratios, achieved state-of-the-art results on the Zero Resource Speech Challenge 2015 tasks. It significantly outperformed previous Siamese network iterations and Correspondence Auto-Encoders (CAE).
As detailed in Figure 5, the model achieved an ABX error rate of 10.4% for within-speaker English tasks. Crucially, the strategies developed on English transferred successfully to Xitsonga, a language from a completely different family (Bantu). This generalization suggests that these sampling principles are universal across human languages.
Conclusion
This research underscores that the effectiveness of Siamese networks for speech is heavily dependent on the sampling distribution. By neutralizing the effects of Zipf’s Law through uniform sampling, researchers can achieve superior phonetic representations even with minimal data. While some generative models like DPGMMs still hold advantages in specific categories, the improvements in ABnet provide a clear path forward for discriminative unsupervised learning. Future work will focus on integrating these improved representations back into Spoken Term Discovery systems to create a virtuous cycle of better data and better models.
Summary updated