Segmental Audio Word2Vec: Representing Utterances as Sequences of Vectors with Applications in Spoken Term Detection

Yu-Hsuan Wang, Hung-yi Lee, Lin-Shan Lee

Abstract

While standard Word2Vec represents text words as semantic vectors, audio Word2Vec maps spoken segments into vectors carrying phonetic information. Traditionally, audio Word2Vec requires pre-defined word boundaries, limiting its utility for raw audio. We propose Segmental Audio Word2Vec, a framework where unsupervised spoken word boundary segmentation and audio Word2Vec are jointly learned and mutually enhanced. This is achieved via a Segmental Sequence-to-Sequence Autoencoder (SSAE) featuring a segmentation gate trained with reinforcement learning. Experiments across English, Czech, French, and German demonstrate that this approach significantly outperforms traditional frame-based methods in spoken term detection.

Introduction

In natural language processing, Word2Vec is a cornerstone for capturing semantic relationships in text. Its speech counterpart, Audio Word2Vec, aims to map spoken segments into a fixed-dimensional space that reflects phonetic structure. However, a significant bottleneck exists: standard Audio Word2Vec models assume that the input audio is already perfectly segmented into individual words.

In real-world scenarios, speech is a continuous stream. Relying on external, disjoint segmentation algorithms is often suboptimal. This paper introduces a framework where segmentation and embedding are learned jointly, allowing the two tasks to mutually enhance each other in a fully unsupervised manner. The result is a model that can listen to a whole sentence, logically partition it into phonetic units, and embed them on the fly.

Methodology: The Segmental Sequence-to-Sequence Autoencoder (SSAE)

The core of this approach is the Segmental Sequence-to-Sequence Autoencoder (SSAE). Unlike standard RNN-based autoencoders, the SSAE incorporates a specialized mechanism to handle continuous audio streams.

The SSAE architecture features an encoder with a segmentation gate and a reset mechanism to produce segment-level embeddings.

The Segmentation Gate and Reset Mechanism

An auxiliary RNN, the Segmentation Gate, is integrated into the encoder. At each audio frame, it makes a binary decision: "pass" or "segment." When a "segment" decision is made, the encoder treats that moment as a word boundary and outputs a fixed-dimensional embedding for the current chunk.

Crucially, the encoder's internal state is reset to zero immediately after a segment boundary. This "Reset Mechanism" ensures that each embedding is derived strictly from the audio within its boundaries, preventing information leakage across segments and forcing the model to capture the complete phonetic profile of the word in a single vector.

Training with Reinforcement Learning

Because the segmentation decisions are discrete and non-differentiable, the authors employ reinforcement learning (specifically Proximal Policy Optimization). The reward function balances two objectives:

  1. Reconstruction Accuracy: Lowering the error when the decoder reconstructs the original audio from the embedding.
  2. Segmentation Penalty: Penalizing the model for creating an excessive number of segments, which prevents it from defaulting to frame-level segmentation.

Experimental Evaluation

The model was evaluated on four languages: English (TIMIT), and Czech, French, and German (GlobalPhone). The primary application was Query-by-Example Spoken Term Detection (STD), where a spoken query is used to find matching instances in an audio database.

The Spoken Term Detection (STD) pipeline using SSAE embeddings for subsequence matching.

Results and Discussion

The SSAE embeddings significantly outperformed traditional frame-based methods like Dynamic Time Warping (DTW). By tying segmentation and embedding together, the tasks improve one another: good boundaries lead to recognizable segments that are easier to reconstruct, which in turn rewards the segmentation gate for making accurate cuts.

Quantitative STD Results (Mean Average Precision): * English: 23.27 (vs. DTW's 12.02) * French: 21.70 (vs. DTW's 11.72) * Czech: 19.41 (vs. DTW's 16.59) * German: 13.82 (vs. DTW's 6.07)

Beyond STD, the model showed strong performance in unsupervised word segmentation, achieving an F1 score of 43.12 on English, surpassing baseline methods like Hierarchical Agglomerative Clustering (33.96).

Conclusion

Segmental Audio Word2Vec provides a robust, unsupervised method for converting continuous speech into meaningful vector sequences. By jointly optimizing for segmentation and reconstruction, the model captures phonetic structures more effectively than disjoint pipelines. Future research may extend these embeddings to tasks such as speech-to-speech translation and automated voice conversion.

Summary updated

Abstract

While Word2Vec represents words (in text) as vectors carrying semantic information, audio Word2Vec was shown to be able to represent signal segments of spoken words as vectors carrying phonetic structure information. Audio Word2Vec can be trained in an unsupervised way from an unlabeled corpus, except the word boundaries are needed. In this paper, we extend audio Word2Vec from word-level to utterance-level by proposing a new segmental audio Word2Vec, in which unsupervised spoken word boundary segmentation and audio Word2Vec are jointly learned and mutually enhanced, so an utterance can be directly represented as a sequence of vectors carrying phonetic structure information. This is achieved by a segmental sequence-to-sequence autoencoder (SSAE), in which a segmentation gate trained with reinforcement learning is inserted in the encoder. Experiments on English, Czech, French and German show very good performance in both unsupervised spoken word segmentation and spoken term detection applications (significantly better than frame-based DTW).
BibTeX
@inproceedings{wang2018segmental,
  author = {Yu-Hsuan Wang and Hung-yi Lee and Lin-Shan Lee},
  title = {{Segmental Audio Word2Vec: Representing Utterances as Sequences of Vectors with Applications in Spoken Term Detection}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=wDtixwDw6WT},
}
Figure