Segmental Audio Word2Vec: Representing Utterances as Sequences of Vectors with Applications in Spoken Term Detection

Yu-Hsuan Wang, Hung-yi Lee, Lin-Shan Lee

Here is a summary of the paper:

TL;DR

This paper introduces Segmental Audio Word2Vec, a model that jointly learns to segment continuous speech into words and represent those segments as fixed-dimensional vectors. By using a Segmental Sequence-to-Sequence Autoencoder (SSAE) trained with reinforcement learning, the model eliminates the need for pre-defined word boundaries. This unsupervised approach significantly outperforms traditional frame-based methods like Dynamic Time Warping (DTW) in spoken term detection tasks across multiple languages.

What Problem Does This Paper Address?

In natural language processing, Word2Vec is famous for turning text words into dense vectors that capture semantic meaning. Researchers have previously adapted this for speech ("Audio Word2Vec") to turn spoken words into vectors that capture phonetic structure. However, there is a major catch: standard Audio Word2Vec requires the audio to already be perfectly segmented into individual words.

In the real world, raw speech is a continuous stream of audio without clear boundaries. While you could use a separate algorithm to guess the word boundaries first and then run Audio Word2Vec, the authors argue this is suboptimal. They propose that segmentation and embedding should be learned jointly so they can mutually enhance each other. The goal is a fully unsupervised model that can listen to a whole sentence, chop it up into logical phonetic units, and embed them on the fly.

Methods

The authors propose a Segmental Sequence-to-Sequence Autoencoder (SSAE). It builds on a standard RNN-based encoder-decoder architecture but introduces a few clever twists:

  • The Segmentation Gate: An extra RNN is inserted into the encoder. At every audio frame, this gate makes a binary decision: "pass" or "segment".
  • The Reset Mechanism: When the gate decides to "segment," it treats that moment as a word boundary. The encoder outputs a fixed-dimensional vector (the embedding) for that chunk of audio. Crucially, the RNN's internal state is then reset to zero. This forces the generated embedding to rely solely on the audio within that specific segment, preventing information from bleeding across word boundaries.
  • Reinforcement Learning (RL): Because the decision to "segment" or "pass" is discrete, standard backpropagation (which requires smooth, differentiable mathematical functions) won't work. The authors use RL (specifically, Proximal Policy Optimization) to train the gate.
  • The Reward Function: The RL agent is rewarded for low reconstruction error (meaning the embeddings accurately capture the audio) but penalized for creating too many segments (to prevent the model from just making every single frame its own segment).

Datasets

The model was trained and evaluated in a completely unsupervised manner on four languages:

  • English: TIMIT corpus
  • Czech, French, German: GlobalPhone corpora

Key Insights and Ideas

  • Joint learning is synergistic: By tying segmentation and embedding together, the tasks improve one another. Good boundaries lead to highly recognizable audio segments, which are easier for the autoencoder to reconstruct. This lower reconstruction error then acts as a reward, teaching the segmentation gate that it made a good cut.
  • The power of the "Reset": Wiping the RNN's memory at every predicted boundary is a simple but powerful architectural choice. It forces the network to pack all the phonetic information of a segment into a single vector, ensuring the embedding is a true representation of that specific spoken word.
  • RL bridges the gap in audio processing: Using reinforcement learning to balance reconstruction accuracy against a penalty for over-segmentation is an elegant way to solve the non-differentiable boundary problem in continuous audio streams.

Key Results

The authors tested the generated embeddings on a Query-by-Example Spoken Term Detection (STD) task—essentially, using a spoken audio clip of a word to search for other instances of that word in a large audio archive.

  • Massive leaps over baselines: SSAE embeddings crushed standard frame-based Dynamic Time Warping (DTW), which is a traditional baseline for this task.
  • Quantitative STD Results (Mean Average Precision):
    • English: 23.27 (vs. DTW's 12.02)
    • French: 21.70 (vs. DTW's 11.72)
    • Czech: 19.41 (vs. DTW's 16.59)
    • German: 13.82 (vs. DTW's 6.07)
  • Segmentation Accuracy: SSAE also outperformed baseline unsupervised segmentation methods like Hierarchical Agglomerative Clustering (HAC). For example, it achieved an F1 segmentation score of 43.12 on English, compared to HAC's 33.96.

Limitations and Future Work

  • Limitations: The model tends to over-segment words into subword units. For instance, in English, the recall for word boundaries was around 50%, but precision was significantly lower, indicating many "extra" cuts. Additionally, the authors noted an anomaly in German where STD performance was high despite mediocre segmentation scores, the exact reason for which remains unclear.
  • Future Work: The authors suggest that this utterance-level Audio Word2Vec could unlock new unsupervised approaches for downstream tasks like speech information summarization, speech-to-speech translation, and voice conversion.

Summary updated

Abstract

While Word2Vec represents words (in text) as vectors carrying semantic information, audio Word2Vec was shown to be able to represent signal segments of spoken words as vectors carrying phonetic structure information. Audio Word2Vec can be trained in an unsupervised way from an unlabeled corpus, except the word boundaries are needed. In this paper, we extend audio Word2Vec from word-level to utterance-level by proposing a new segmental audio Word2Vec, in which unsupervised spoken word boundary segmentation and audio Word2Vec are jointly learned and mutually enhanced, so an utterance can be directly represented as a sequence of vectors carrying phonetic structure information. This is achieved by a segmental sequence-to-sequence autoencoder (SSAE), in which a segmentation gate trained with reinforcement learning is inserted in the encoder. Experiments on English, Czech, French and German show very good performance in both unsupervised spoken word segmentation and spoken term detection applications (significantly better than frame-based DTW).
BibTeX
@inproceedings{wang2018segmental,
  author = {Yu-Hsuan Wang and Hung-yi Lee and Lin-Shan Lee},
  title = {{Segmental Audio Word2Vec: Representing Utterances as Sequences of Vectors with Applications in Spoken Term Detection}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=wDtixwDw6WT},
}

Loading related content...

Figure