Neural Network Based End-to-End Query by Example Spoken Term Detection

Dhananjay Ram, Lesly Miculicich, Hervé Bourlard

TL;DR

This paper introduces a novel end-to-end neural network architecture for "query by example" spoken term detection (QbE-STD). By jointly optimizing feature extraction and matching, their system significantly outperforms traditional methods, particularly in challenging zero-resource scenarios.

What Problem Does This Paper Address?

Imagine you have a massive archive of audio recordings – think news broadcasts, radio shows, or even user-generated content – and you want to find specific spoken segments within it. Traditional search engines rely on text, which might be missing or inaccurate for audio. "Query by example" spoken term detection (QbE-STD) aims to solve this by allowing users to provide a spoken query (e.g., a short audio clip of a name or phrase) and find all occurrences of that query in a large audio collection. This is particularly challenging in "zero-resource" scenarios where you don't have pre-existing language models, phonetic dictionaries, or extensive labeled data for the target languages. Existing state-of-the-art methods often rely on complex pipelines involving feature extraction (like bottleneck features from deep neural networks) and template matching (like Dynamic Time Warping - DTW). While effective, these methods often treat feature extraction and matching as separate, independently optimized steps, limiting overall performance.

Methods

The core contribution of this paper is an end-to-end neural network architecture for QbE-STD. This approach integrates the feature extraction and matching stages into a single, jointly trainable model.

Here's a breakdown of their methodology:

  • Representation Learning: They explore both monolingual and, crucially, multilingual bottleneck features. The multilingual approach uses a shared network trained on multiple languages simultaneously. This "multitask learning" principle encourages the network to learn language-independent representations, which are more robust to variations across languages. They found that increasing the number of languages in the multilingual training improved performance.
  • CNN-based Matching: Instead of relying solely on DTW for matching, they propose treating the similarity matrix between query and test utterance features as an "image." A Convolutional Neural Network (CNN) is then trained to classify these similarity matrices, identifying patterns indicative of a query match. This CNN-based approach offers a more discriminative learning framework compared to DTW's frame-by-frame decisions.
  • End-to-End Architecture: The key innovation is combining the multilingual bottleneck feature extractor and the CNN-based matching module into a single neural network. This allows for joint optimization of both representation learning and the matching process. The input is raw spectral features (MFCCs), and the output is a score indicating the likelihood of the query occurring. They also demonstrate that the CNN matching block can be used as a "loss function" to further fine-tune the feature extractor.

Datasets

The authors evaluate their methods on two challenging multilingual datasets:

  • Spoken Web Search (SWS) 2013: This dataset is used for training and evaluating the CNN-based matching and the end-to-end systems. It comprises speech data from 9 low-resource languages.
  • Query by Example Search on Speech Task (QUESST) 2014: This dataset is used to evaluate the generalization ability of their proposed models. It includes data from 6 languages.

Key Insights and Ideas

  • Multilingual Features are Key: Training bottleneck feature extractors on multiple languages significantly outperforms monolingual approaches, especially as more languages are included. This highlights the power of learning language-agnostic representations.
  • CNNs Excel at Pattern Recognition: Treating the similarity matrix as an image and using a CNN for classification proves more effective than traditional DTW for identifying query occurrences. This is because CNNs can capture more complex, global patterns.
  • End-to-End Learning Boosts Performance: Jointly training the feature extraction and matching components leads to substantial improvements over separate training. This allows the system to learn features that are specifically optimized for the matching task.
  • CNN as a Loss Function: Using the trained CNN as a loss function to fine-tune the feature extractor is a clever way to guide the learning of even better, more discriminative features.

Key Results

The proposed end-to-end system consistently outperforms both the baseline DTW-based matching and the CNN-based matching approaches across both SWS 2013 and QUESST 2014 datasets.

  • Significant Improvement: The end-to-end system achieves significant improvements in terms of both minimum normalized cross-entropy (minCnxe) and maximum Term Weighted Value (MTWV) scores.
  • Generalizability: The CNN-based matching and end-to-end systems demonstrate good generalization capabilities to the QUESST 2014 dataset, even when trained on SWS 2013.
  • Language-Specific Performance: While overall performance is strong, there are variations in performance across different languages, particularly for low-resource languages in SWS 2013, which the authors attribute to limited training data.

Limitations and Future Work

  • Limited Data for Low-Resource Languages: The performance gains are not uniform across all languages, especially for those with very little training data. This suggests that further research into low-resource language modeling and data augmentation techniques could be beneficial.
  • Variable Input Size: Handling variable-sized similarity matrices for CNN input required specific down-sampling and up-sampling strategies. While effective, exploring more sophisticated methods for variable-length inputs could be a future direction.
  • Unsupervised Unit Discovery: The authors suggest that the CNN matching block, when used as a loss function, can be valuable for other tasks like unsupervised unit discovery, indicating potential broader applications of their work.

Summary updated

Abstract

This paper focuses on the problem of query by example spoken term detection (QbE-STD) in zero-resource scenario. State-of-the-art approaches primarily rely on dynamic time warping (DTW) based template matching techniques using phone posterior or bottleneck features extracted from a deep neural network (DNN). We use both monolingual and multilingual bottleneck features, and show that multilingual features perform increasingly better with more training languages. Previously, it has been shown that the DTW based matching can be replaced with a CNN based matching while using posterior features. Here, we show that the CNN based matching outperforms DTW based matching using bottleneck features as well. In this case, the feature extraction and pattern matching stages of our QbE-STD system are optimized independently of each other. We propose to integrate these two stages in a fully neural network based end-to-end learning framework to enable joint optimization of those two stages simultaneously. The proposed approaches are evaluated on two challenging multilingual datasets: Spoken Web Search 2013 and Query by Example Search on Speech Task 2014, demonstrating in each case significant improvements.
BibTeX
@article{ram2020neural,
  author = {Dhananjay Ram and Lesly Miculicich and Hervé Bourlard},
  title = {{Neural Network Based End-to-End Query by Example Spoken Term Detection}},
  journal = {IEEE/ACM Trans. ASLP},
  year = {2020},
  doi = {10.1109/taslp.2020.2988788},
}
Figure