TL;DR
This paper explores how pre-trained self-supervised speech models, like HuBERT and Wav2Vec 2.0, can be used to create acoustic word embeddings (AWEs) without requiring any task-specific training. They demonstrate that simple pooling methods applied to these representations can achieve state-of-the-art or near state-of-the-art performance on English AWE tasks, and surprisingly, even outperform multilingual models on other languages.
What Problem Does This Paper Address?
Traditional methods for creating acoustic word embeddings (AWEs) – fixed-size vector representations of spoken words – often rely on complex, supervised, or computationally intensive unsupervised learning techniques. These methods typically require either labeled data or a separate unsupervised term detection (UTD) step, which can be challenging, especially for low-resource languages where such resources are scarce. The paper addresses the question of whether powerful, pre-trained self-supervised speech models, which have shown great success in other speech tasks, can simplify AWE creation by implicitly encoding the necessary sequential information within their representations, thus bypassing the need for explicit sequential modeling or UTD.
Methods
The core of the approach involves using pre-trained self-supervised speech models as feature extractors. Specifically, the authors experiment with HuBERT (HB) and Wav2Vec 2.0 (W2V2), including a multilingual version (XLSR-53). The process involves:
- Encoding: Feeding raw audio segments into the pre-trained models to obtain contextualized frame-level representations.
- Pooling: Applying simple pooling functions (subsampling, argmax, sum, and mean) to these frame-level representations to derive fixed-dimensional AWEs. The paper hypothesizes that contextualized representations might make simple pooling sufficient.
- Evaluation: Assessing the quality of the generated AWEs using a standard same-different word discrimination task, measured by Average Precision (AP).
They also investigate the effect of frame-level normalization and analyze representations from different layers of the Transformer models. For cross-lingual evaluation, they use the same pooling strategies but apply them to models pre-trained on English or multiple languages.
Datasets
- Training: The pre-trained models (HuBERT Large, Wav2Vec 2.0 Large, XLSR-53) were pre-trained on large datasets like Libri-light (60k hours of read speech).
- Evaluation:
- English: Buckeye corpus (conversational speech), Librispeech (read speech).
- Non-English: Xitsonga (NCHLT corpus - read speech), Mandarin and French (ZeroSpeech Challenge 2017 datasets - read speech).
Key Insights and Ideas
- Context is King: Self-supervised models like HuBERT implicitly encode significant sequential information within their contextualized representations, making simple pooling methods surprisingly effective for AWEs.
- HuBERT Shines: HuBERT representations, particularly with mean-pooling, demonstrate superior performance compared to Wav2Vec 2.0, even when both are trained on English. This suggests the pre-training objective plays a crucial role.
- Cross-Lingual Surprise: English-trained HuBERT representations perform remarkably well on non-English languages (Xitsonga, Mandarin, French), often outperforming multilingual Wav2Vec 2.0 models, despite not being trained on those languages. This highlights the transferability of learned acoustic features.
- Pooling Simplicity: Mean-pooling, a computationally inexpensive method, can rival or even surpass more complex methods like subsampling (which explicitly models sequence) when applied to powerful self-supervised representations, especially when dimensionality is controlled.
- Language-Specific Nuances: While English-trained models transfer well, the paper hints that the explicit sequential modeling of subsampling might still be beneficial for languages with significantly different phonotactic patterns, as English context doesn't perfectly generalize.
Key Results
- On the English Buckeye development set, HuBERT with mean-pooling achieved an AP of 65.3%, rivaling state-of-the-art learned pooling models with equivalent dimensionality.
- For Xitsonga, mean-pooled HuBERT representations achieved an AP of 39.0% (1024 dims), outperforming subsampling (46.0% at 10240 dims) when dimensionality is considered, and performing competitively with previous unsupervised AWE models.
- In cross-lingual settings, multilingual Wav2Vec 2.0 performed better than English Wav2Vec 2.0, but English HuBERT consistently outperformed both.
- Qualitative analysis using t-SNE visualizations showed that HuBERT AWEs successfully clustered instances of words with similar phone sequences but different orders, indicating encoding of sequential information. Further analysis suggested that HuBERT might even capture word-level context beyond immediate coarticulation.
Limitations and Future Work
- Dimensionality Trade-off: While mean-pooling is efficient, reducing dimensionality via PCA for comparison with prior work led to performance drops, particularly for English. The paper notes that for non-English languages, reducing dimensionality before subsampling had less impact.
- Cross-Lingual Generalization Limits: The English-trained HuBERT model's performance on non-English languages, while strong, still lags behind what might be achieved with a model specifically trained on those languages.
- Future Directions: The authors suggest exploring continued self-supervised training of models like HuBERT on small amounts of unlabeled target language data to further improve cross-lingual performance. They also ponder whether the observed word-level context encoding in AWEs is always desirable for downstream tasks.
Summary updated