Introduction
The Zero Resource Speech Challenge (ZeroSpeech) 2021 focuses on developing speech processing systems for languages lacking textual or linguistic resources. The primary objective is to learn phonetic, lexical, semantic, and syntactic structures directly from raw audio. Traditional approaches often utilize Contrastive Predictive Coding (CPC) to extract features, which are then clustered into discrete "pseudo-phones." However, these features frequently contain "distractor" information, such as speaker identity, which hinders the performance of downstream linguistic models.
The University of Wroclaw’s submission demonstrates that information-retrieval-inspired methods can effectively refine these raw acoustic features. By removing speaker-specific information and employing "fuzzy" pattern matching, the team significantly narrowed the gap between raw audio and high-level linguistic understanding without the use of transcribed text.
Methodology
The proposed system extends the standard CPC-based baseline through several low-resource techniques:
- Speaker-Invariant Embeddings: To enhance phonetic tasks, CPC embeddings were projected into the nullspace of a linear speaker classifier. This process mathematically isolates and removes speaker-specific information while preserving the underlying phonetic content.
- Centroid Averaging: Embeddings were denoised by calculating a weighted average of each vector with its corresponding cluster centroid, stabilizing representations for subsequent pattern matching.
- Fuzzy Dictionary Lookup: For lexical tasks, the team implemented a dictionary lookup using Dynamic Time Warping (DTW). Instead of comparing discrete pseudo-phone IDs, they calculated distances between underlying centroid vectors, facilitating a "soft" matching approach.
- Unsupervised Word Discovery: Semantic relationships were captured by using SentencePiece to discover "pseudo-words" from pseudo-phone streams. A Word2Vec model was then trained on these discovered units.
Experimental Setup
The models were evaluated on the ZeroSpeech 2021 benchmarks using the Libri-Light and LibriSpeech datasets. Evaluation tasks included:
- ABX (Phonetic): Phonetic discrimination error rates.
- sWUGGY (Lexical): Distinguishing real words from phonologically plausible non-words.
- sSIMI (Semantic): Judging similarity between spoken words.
- sBLIMP (Syntactic): Identifying grammatically correct sentences.
Results and Analysis
The integration of nullspace projection and centroid averaging led to a notable reduction in error rates across phonetic tasks. As shown in Figure 3, optimizing nullspace dimensionality directly impacts ABX performance, with the 448-dimensional nullspace yielding the lowest error rates in several categories.
In the lexical domain, the DTW-based lookup achieved a classification accuracy of 73.92% on the base set. The team found that "sharpening" the distance matrix (e.g., raising it to a power of 1.6) improved results by making the model more sensitive to correct matches and more punishing toward errors.
Key insights from the evaluation include:
- Speaker Identity as Noise: Explicitly removing speaker information via nullspace projection significantly boosts phonetic recognition, confirming that unsupervised models often overfit to speaker characteristics rather than phonetic content.
- Stability of CPC: While CPC features may be noisy for training complex language models from scratch, they are highly effective for pattern matching and retrieval tasks.
- Semantic Discovery: The use of SentencePiece and Word2Vec on unsupervised units allowed the model to achieve the best score in the LibriSpeech subcategory for semantic similarity, outperforming provided RoBERTa-based toplines.
Conclusion
The University of Wroclaw’s submission highlights the power of simple, robust information retrieval techniques in unsupervised speech processing. While the system performed exceptionally well on natural speech, it faced challenges with synthesized audio, indicating a reliance on specific acoustic patterns. Furthermore, syntactic tasks (sBLIMP) remain a significant challenge, with the team's LSTM model achieving 53% accuracy—only a slight improvement over the baseline. Future work will explore integrating centroid-based denoising directly into the CPC training process as a loss function.
Summary updated