Temporally-Aware Acoustic Unit Discovery for Zerospeech 2019 Challenge

Bolaji Yusuf, Alican Gök, Batuhan Gündogdu, Oyku Deniz Kose, Murat Saraclar

Abstract

This paper presents two unsupervised approaches for acoustic unit discovery in the context of the ZeroSpeech 2019 Challenge. By incorporating temporal continuity priors into a Self-Organizing Map (SOM) and a Correspondence Recurrent Sparse Autoencoder (CoRSA), the proposed methods effectively extract sub-word units from raw audio without textual supervision. Experimental results on the Indonesian "surprise" language demonstrate significant improvements in signal discriminability (ABX error) and data compression (bitrate) compared to the challenge baseline, highlighting the importance of temporal awareness in unsupervised speech processing.

Introduction

Modern Automatic Speech Recognition (ASR) systems typically require thousands of hours of transcribed audio. This "data-hungry" nature poses a significant barrier for the majority of the world's 7,000+ languages, many of which are low-resource or lack a standard writing system. The ZeroSpeech 2019 Challenge addresses this by tasking researchers to build speech systems using zero text labels, necessitating the automatic discovery of acoustic units (e.g., phonemes) directly from raw audio. A primary challenge in this domain is that standard clustering techniques often ignore the temporal sequentiality of speech, where neighboring frames are highly likely to belong to the same acoustic unit.

Methodology

The authors propose two distinct architectures designed to prioritize temporal awareness during the discovery process.

SOM-RNN Hybrid

The first approach utilizes a Temporally Biased SOM. Unlike standard SOMs that cluster individual frames, this model updates clusters using a weighted average of neighboring frames, ensuring that the resulting units represent stable acoustic events. To further refine these units, "Unit Pooling" is employed to map context-dependent sounds into broader, context-independent categories. Finally, a Gated Recurrent Unit (GRU) network is trained to predict these pooled labels from raw audio features.

Correspondence Recurrent Sparse Autoencoder (CoRSA)

The second approach, CoRSA, leverages a GRU-based autoencoder to compress speech into a bottleneck layer.

The CoRSA architecture utilizes GRU layers and correspondence training to learn robust acoustic units.

Key components of CoRSA include:

  • Sparsity Constraints: A negative L2-norm penalty forces the network to select a single "unit" at each time step, resulting in a sequence of one-hot vectors.
  • Correspondence Training: The model is fine-tuned using pairs of similar audio segments identified via Unsupervised Term Discovery (UTD). By learning to reconstruct one segment from its "twin," the model learns to ignore speaker-specific noise and focus on linguistic content.
  • Temporal Filtering: A median filter is applied to the output to smooth state assignments, which significantly reduces the bitrate by eliminating high-frequency "jitter."

Experimental Results

The models were evaluated on the ZeroSpeech 2019 datasets, including English (development) and Indonesian (surprise test language). Performance was measured using the ABX error rate (discriminability) and Bitrate (compression).

Comparison of ABX error and bitrate across different systems on the Indonesian surprise language.

As shown in the results, the CoRSA system achieved an ABX error of 10.87% on the Indonesian test set, a substantial improvement over the 27.46% baseline. Furthermore, the CoRSA system achieved a bitrate of 29.46, which is lower than both the baseline (74.55) and the supervised "Topline" system (35.20). While the discovered units proved highly discriminative, the Mean Opinion Score (MOS) for synthesized speech remained comparable to the baseline, indicating that while unit discovery has improved, high-quality speech reconstruction remains a challenge.

Conclusion

The integration of temporal continuity as a prior—whether through biased SOM updates or post-processing filters—is crucial for effective acoustic unit discovery. The CoRSA approach, in particular, demonstrates that correspondence learning and temporal smoothing can yield units that are both highly discriminative and efficient. Future work will focus on integrating these units with advanced neural vocoders to improve synthesis quality.

Summary updated

Abstract

Zero-resource speech processing efforts focus on unsupervised discovery of sub-word acoustic units. Common approaches work with spatial similarities between the acoustic frame representations within Bayesian or neural network-based frameworks. We propose two methods that utilize the temporal proximity information in addition to the acoustic similarity for clustering frames into acoustic units. The first approach uses a temporally biased self-organizing map (SOM) to discover such units. Since the SOM unit indices are correlated with (vector) spatial distance, we pool neighboring units and then train a recurrent neural network to predict each pooled unit. The second approach incorporates temporal awareness by training a recurrent sparse autoencoder, in which unsupervised clustering is done on the intermediate softmax layer. This network is then fine-tuned using aligned pairs of acoustically similar sequences obtained via unsupervised term discovery. Our approaches outperform the provided baseline system on two main metrics of the Zerospeech 2019 challenge, ABX-discriminability and bitrate of the quantized embeddings, both for English and the surprise language. Furthermore, the temporal-awareness and the post-filtering techniques adopted in this work resulted in an enhanced continuity of the decoding, yielding low bitrates.
BibTeX
@inproceedings{yusuf2019temporally,
  author = {Bolaji Yusuf and Alican Gök and Batuhan Gündogdu and Oyku Deniz Kose and Murat Saraclar},
  title = {{Temporally-Aware Acoustic Unit Discovery for Zerospeech 2019 Challenge}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-1430},
}
Figure