Query-by-example Spoken Term Detection using Attention-based Multi-hop Networks

Chia-Wei Ao, Hung-yi Lee

Abstract

Retrieving spoken content with spoken queries, or query-by-example spoken term detection (STD), is attractive because it makes possible the matching of signals directly on the acoustic level without transcribing them into text. Here, we propose an end-to-end query-by-example STD model based on an attention-based multi-hop network, whose input is a spoken query and an audio segment containing several utterances; the output states whether the audio segment includes the query. The model can be trained in either a supervised scenario using labeled data, or in an unsupervised fashion. In the supervised scenario, we find that the attention mechanism and multiple hops improve performance, and that the attention weights indicate the time span of the detected terms. In the unsupervised setting, the model mimics the behavior of the existing query-by-example STD system, yielding performance comparable to the existing system but with a lower search time complexity.

Introduction

Spoken Term Detection (STD) aims to locate specific words or phrases within large audio databases. In the "Query-by-Example" (QbE) variant, the search is performed using a spoken snippet rather than text. This approach is vital for low-resource languages lacking robust speech-to-text systems or for hands-free applications where transcription is unreliable.

Traditionally, Dynamic Time Warping (DTW) has been the standard for QbE-STD. However, DTW suffers from high computational complexity ($O(M \times N)$) and a lack of learnable parameters. This paper, presented at ICASSP 2018, proposes an end-to-end neural network that leverages attention mechanisms and multi-hop iterations to overcome these limitations.

Training Paradigms

The proposed model is versatile, supporting both supervised and unsupervised training environments to accommodate different data availability scenarios.

Overview of the proposed system and its training paradigms: supervised learning with binary labels and unsupervised learning using a teacher-student approach.
  • Supervised Scenario: The model is trained on labeled pairs where the ground truth indicates whether a specific query exists within an audio segment (e.g., label 1 for existence, 0 for absence).
  • Unsupervised Scenario (Teacher-Student): To handle data without manual transcriptions, the model acts as a "student" mimicking a "teacher" (a traditional DTW system). The network learns to approximate the DTW scores, inheriting its matching capabilities while gaining the inference speed of a neural network.

Proposed Architecture

The core of the system is an Attention-based Multi-hop Network designed to replace rigid signal matching with flexible, learnable representations.

Detailed architecture of the network showing (A) the query encoder and (B) the audio segment encoder with the attention mechanism.
  1. Encoding: Both the spoken query and the target audio segment are processed through Long Short-Term Memory (LSTM) networks. The query is compressed into a fixed-length vector ($V_Q$), while the audio segment is represented as a sequence of hidden states ($S_1, \dots, S_8$).
  2. Attention Mechanism: The model calculates attention weights ($\alpha_i$) by comparing the query vector $V_Q$ with each audio state $S_i$. These weights identify the most relevant parts of the audio, which are then aggregated into a "story vector" ($V_S$).
  3. Multi-hop Iterations: The model performs multiple "hops." In each hop, the query vector is updated with information from the story vector (akin to query expansion). This iterative process allows the model to refine its focus, effectively filtering out noise and improving detection accuracy.

Experimental Results

The researchers evaluated the model using the LibriSpeech corpus. In supervised tests, the attention-based model achieved a Mean Average Precision (MAP) of 0.65–0.67, significantly outperforming the DTW baseline of 0.61.

Crucially, in the unsupervised setting, the model matched the performance of DTW while being 7 times faster during inference. This speedup is due to the reduction in complexity from $O(M \times N)$ to $O(M \times n)$, where $n$ is the small, fixed number of hops (typically 3).

Conclusion and Future Work

The attention-based multi-hop network represents a significant step forward for QbE-STD, offering superior accuracy in supervised settings and dramatic efficiency gains in unsupervised ones. While the model currently faces challenges with entirely unseen words (the "out-of-vocabulary" problem), future work will focus on improving generalization and directly predicting the precise time spans of detected terms.

Summary updated

Abstract

Retrieving spoken content with spoken queries, or query-by- example spoken term detection (STD), is attractive because it makes possible the matching of signals directly on the acoustic level without transcribing them into text. Here, we propose an end-to-end query-by-example STD model based on an attention-based multi-hop network, whose input is a spoken query and an audio segment containing several utterances; the output states whether the audio segment includes the query. The model can be trained in either a supervised scenario using labeled data, or in an unsupervised fashion. In the supervised scenario, we find that the attention mechanism and multiple hops improve performance, and that the attention weights indicate the time span of the detected terms. In the unsupervised setting, the model mimics the behavior of the existing query-by-example STD system, yielding performance comparable to the existing system but with a lower search time complexity.
BibTeX
@inproceedings{ao2018query,
  author = {Chia-Wei Ao and Hung-yi Lee},
  title = {{Query-by-example Spoken Term Detection using Attention-based Multi-hop Networks}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=0pKUjUmjik},
}
Figure