Abstract
While standard Word2Vec represents text words as semantic vectors, audio Word2Vec maps spoken segments into vectors carrying phonetic information. Traditionally, audio Word2Vec requires pre-defined word boundaries, limiting its utility for raw audio. We propose Segmental Audio Word2Vec, a framework where unsupervised spoken word boundary segmentation and audio Word2Vec are jointly learned and mutually enhanced. This is achieved via a Segmental Sequence-to-Sequence Autoencoder (SSAE) featuring a segmentation gate trained with reinforcement learning. Experiments across English, Czech, French, and German demonstrate that this approach significantly outperforms traditional frame-based methods in spoken term detection.
Introduction
In natural language processing, Word2Vec is a cornerstone for capturing semantic relationships in text. Its speech counterpart, Audio Word2Vec, aims to map spoken segments into a fixed-dimensional space that reflects phonetic structure. However, a significant bottleneck exists: standard Audio Word2Vec models assume that the input audio is already perfectly segmented into individual words.
In real-world scenarios, speech is a continuous stream. Relying on external, disjoint segmentation algorithms is often suboptimal. This paper introduces a framework where segmentation and embedding are learned jointly, allowing the two tasks to mutually enhance each other in a fully unsupervised manner. The result is a model that can listen to a whole sentence, logically partition it into phonetic units, and embed them on the fly.
Methodology: The Segmental Sequence-to-Sequence Autoencoder (SSAE)
The core of this approach is the Segmental Sequence-to-Sequence Autoencoder (SSAE). Unlike standard RNN-based autoencoders, the SSAE incorporates a specialized mechanism to handle continuous audio streams.
The Segmentation Gate and Reset Mechanism
An auxiliary RNN, the Segmentation Gate, is integrated into the encoder. At each audio frame, it makes a binary decision: "pass" or "segment." When a "segment" decision is made, the encoder treats that moment as a word boundary and outputs a fixed-dimensional embedding for the current chunk.
Crucially, the encoder's internal state is reset to zero immediately after a segment boundary. This "Reset Mechanism" ensures that each embedding is derived strictly from the audio within its boundaries, preventing information leakage across segments and forcing the model to capture the complete phonetic profile of the word in a single vector.
Training with Reinforcement Learning
Because the segmentation decisions are discrete and non-differentiable, the authors employ reinforcement learning (specifically Proximal Policy Optimization). The reward function balances two objectives:
- Reconstruction Accuracy: Lowering the error when the decoder reconstructs the original audio from the embedding.
- Segmentation Penalty: Penalizing the model for creating an excessive number of segments, which prevents it from defaulting to frame-level segmentation.
Experimental Evaluation
The model was evaluated on four languages: English (TIMIT), and Czech, French, and German (GlobalPhone). The primary application was Query-by-Example Spoken Term Detection (STD), where a spoken query is used to find matching instances in an audio database.
Results and Discussion
The SSAE embeddings significantly outperformed traditional frame-based methods like Dynamic Time Warping (DTW). By tying segmentation and embedding together, the tasks improve one another: good boundaries lead to recognizable segments that are easier to reconstruct, which in turn rewards the segmentation gate for making accurate cuts.
Quantitative STD Results (Mean Average Precision): * English: 23.27 (vs. DTW's 12.02) * French: 21.70 (vs. DTW's 11.72) * Czech: 19.41 (vs. DTW's 16.59) * German: 13.82 (vs. DTW's 6.07)
Beyond STD, the model showed strong performance in unsupervised word segmentation, achieving an F1 score of 43.12 on English, surpassing baseline methods like Hierarchical Agglomerative Clustering (33.96).
Conclusion
Segmental Audio Word2Vec provides a robust, unsupervised method for converting continuous speech into meaningful vector sequences. By jointly optimizing for segmentation and reconstruction, the model captures phonetic structures more effectively than disjoint pipelines. Future research may extend these embeddings to tasks such as speech-to-speech translation and automated voice conversion.
Summary updated