CNN-Based Spoken Term Detection and Localization without Dynamic Programming

Tzeviya Sylvia Fuchs, Yael Segal, Joseph Keshet

TL;DR

This paper introduces EmbeddingSpeechYOLO (ESY), a novel CNN-based system for spoken term detection and localization that bypasses traditional dynamic programming. ESY simultaneously predicts term occurrences and their locations by learning to map audio segments to phonetic embeddings, enabling efficient detection of both known and unknown words without task-specific training.

What Problem Does This Paper Address?

Spoken Term Detection (STD) aims to find specific words or phrases within audio recordings. Traditional methods often rely on dynamic programming (like dynamic time warping) to align and match target terms to the audio signal. While effective, these methods can be computationally intensive, especially for long audio segments. Furthermore, detecting terms that were not explicitly seen during training (Out-Of-Vocabulary or OOV terms) remains a significant challenge. This paper tackles these limitations by proposing a system that can detect and locate both in-vocabulary (IV) and out-of-vocabulary (OOV) terms efficiently, without the need for dynamic programming, by leveraging learned phonetic embeddings.

Methods

The core of the proposed system, ESY, is built upon the SpeechYOLO architecture, which was originally designed for object detection in images and adapted for keyword spotting. ESY extends this by treating audio segments as "images" and predicting "objects" (terms) within them.

Key methodological innovations include:

  • Embedding-based Detection: Instead of direct acoustic matching, ESY predicts word embeddings for segments of the audio. These predicted embeddings are then compared to the embedding of the target term. This approach allows for the detection of terms not seen during training by relying on their phonetic similarity captured in the embedding space.
  • Simultaneous Prediction: ESY divides the input audio into fixed-size "cells." Each cell is responsible for detecting a single event (term) and simultaneously predicts:
  • A proposed embedding vector for the audio segment.
  • The center time of the term within the cell.
  • The duration of the term. This parallel processing eliminates the need for sequential dynamic programming.

  • Phonetic Embeddings for OOV Terms: To handle OOV terms, ESY utilizes their phonetic representations. These phonetic sequences are converted into embeddings using a pre-trained embedding function, allowing the model to generalize to unseen words based on their sound structure.

  • Multi-task Loss Function: The system is trained with a loss function that combines three objectives:
  • Maximizing the cosine similarity between the predicted and target term embeddings (for detection).
  • Minimizing similarity when a term is not present (to reduce false positives).
  • Minimizing the error in predicting the term's center time and duration (for localization).

Datasets

The paper evaluates ESY on three datasets:

  • Switchboard: A corpus of English conversational telephone speech. Used for single-word setup experiments.
  • LibriSpeech: Derived from read audiobooks, this dataset was used for multi-word setup experiments. It was split into training, validation, and test sets, with specific attention to creating IV and OOV terms for the test and validation sets.
  • TIMIT: Used for testing purposes, particularly to evaluate performance on OOV terms and compare localization accuracy against a forced aligner.

Key Insights and Ideas

  • Phonetic Embeddings as a Bridge: The use of phonetic embeddings is crucial for generalizing to OOV terms. By representing words based on their sound structure rather than just their acoustic realization, the model can infer the presence of unseen words.
  • Trading Computation for Model Complexity: The authors explicitly trade the time complexity of dynamic programming for the size and depth of the neural network. This suggests a shift towards end-to-end deep learning models that can learn complex temporal patterns implicitly.
  • The Power of "Soft" Penalties: The paper highlights that using a "softer" dis-similarity function (like |0 - cos²(e1, e2)|) for cases where a term is not present leads to better performance. This is attributed to the fact that the model doesn't know what sound is present, so a less aggressive penalty for a mismatch is more robust.
  • Localization is Achievable Without Explicit Alignment: ESY demonstrates that accurate localization of spoken terms can be achieved by jointly predicting temporal centers and durations, without relying on explicit phonetic or word-level alignment during inference.

Key Results

  • Single-Word Detection: On the Switchboard dataset, ESY achieved competitive results, though slightly lower than the Multi-View algorithm used for generating embeddings. However, ESY significantly outperformed Siamese CNN and LSTM-based approaches in terms of Average Precision (AP).
  • Multi-Word Detection: ESY outperformed adapted versions of the Multi-View algorithm (Multi-View-QbE and Multi-View-STD) on the LibriSpeech dataset for both IV and OOV terms. ESY achieved the highest AP values in this setup.
  • Localization Accuracy: ESY demonstrated high Intersection-Over-Union (IOU) values for localization, indicating precise temporal boundaries, even for OOV terms. When compared to the Montreal Forced Aligner (MFA) on TIMIT, ESY's IOU was 0.73 compared to MFA's 0.9, which is considered a strong performance given MFA's access to full transcriptions.
  • OOV Performance: The system showed promising results on OOV terms, with AP values that were often only slightly lower than for IV terms, especially when using the |0 - cos²(e1, e2)| dis-similarity function.

Limitations and Future Work

  • Performance Gap with Embedding Generator: In the single-word setup, ESY's performance was bounded by the quality of the embeddings generated by the Multi-View algorithm. Future work could explore end-to-end training of both the embedding generation and detection components.
  • OOV AP Improvement: While ESY performs well on OOV terms, there's still room for improvement in AP values for these terms.
  • Combining IV and OOV Detection: The authors suggest combining the SpeechYOLO algorithm (presumably for IV terms) with the ESY algorithm (for OOV terms) as a future direction.
  • Generalization to Noisy Environments: The current evaluation is primarily on read speech corpora. Testing and improving performance in more challenging, noisy, or conversational speech scenarios would be a valuable next step.

Summary updated

Abstract

In this paper, we propose a spoken term detection algorithm for simultaneous prediction and localization of in-vocabulary and out-of-vocabulary terms within an audio segment. The proposed algorithm infers whether a term was uttered within a given speech signal or not by predicting the word embeddings of various parts of the speech signal and comparing them to the word embedding of the desired term. The algorithm utilizes an existing embedding space for this task and does not need to train a task-specific embedding space. At inference the algorithm simultaneously predicts all possible locations of the target term and does not need dynamic programming for optimal search. We evaluate our system on several spoken term detection tasks on read speech corpora.
BibTeX
@inproceedings{fuchs2021cnn,
  author = {Tzeviya Sylvia Fuchs and Yael Segal and Joseph Keshet},
  title = {{CNN-Based Spoken Term Detection and Localization without Dynamic Programming}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=Ab0PJ500On},
}
Figure