Unsupervised Spoken Term Detection with Spoken Queries by Multi-level Acoustic Patterns with Varying Model Granularity

Cheng-Tao Chung, Chun-an Chan, Lin-Shan Lee

Abstract

This paper introduces a "zero-resource" approach for searching spoken words in audio archives using only spoken queries, eliminating the need for manual transcriptions or text-based models. The core innovation lies in the discovery of multi-level acoustic patterns across a three-dimensional granularity space—temporal, phonetic, and acoustic. By combining these diverse patterns and utilizing a soft similarity measure based on KL-divergence, the proposed system achieves a 16.16% absolute improvement in Mean Average Precision (MAP) over traditional frame-based Dynamic Time Warping (DTW) while significantly reducing computational overhead.

Introduction

Spoken Term Detection (STD) is critical for navigating large audio archives. While traditional systems rely on Automatic Speech Recognition (ASR), such methods are often inapplicable to the thousands of "low-resource" languages that lack annotated training data. In these scenarios, researchers typically turn to unsupervised methods like frame-based Dynamic Time Warping (DTW). However, frame-level DTW is highly sensitive to speaker variability and noise, and its frame-by-frame comparison is computationally prohibitive for large-scale applications.

This research proposes a method to bridge the gap between simple frame-level matching and complex ASR by automatically discovering a "vocabulary" of acoustic patterns from unlabeled audio, allowing for more robust and efficient search.

Methodology: The Granularity Space

The fundamental concept of this work is the use of Multi-level Acoustic Patterns with Varying Model Granularity. Rather than seeking a single optimal set of pseudo-phonemes, the system discovers multiple sets of patterns across a three-dimensional space:

  1. Temporal Granularity ($m$): Defined by the number of states in the Hidden Markov Model (HMM), representing the duration of a pattern.
  2. Phonetic Granularity ($n$): The total number of distinct patterns discovered (vocabulary size).
  3. Acoustic Granularity ($l$): The number of Gaussians per state, capturing acoustic variation.
The 3D Granularity Space defined by temporal (m), phonetic (n), and acoustic (l) parameters.

Workflow and Representation

The system clusters audio segments into patterns through an iterative, unsupervised HMM training process. Spoken documents and queries are then represented as sequences of these pattern indices. To enhance robustness, the authors employ Soft Similarity using KL-divergence between the underlying HMMs of different patterns, rather than relying on hard index matching.

Matching is performed using either diagonal sub-sequence matching or a pattern-based DTW. Because these methods operate on patterns spanning multiple frames, the search process is significantly faster than standard frame-level DTW.

Experimental Setup

The approach was evaluated using the TIMIT corpus. The training set served as the unlabeled archive, while 32 spoken words from the test set were used as queries. The primary evaluation metric was Mean Average Precision (MAP).

Results and Discussion

The experimental results highlight the power of diversity in acoustic modeling. The baseline frame-based DTW achieved a MAP of 10.16%. In contrast, the proposed method demonstrated that patterns discovered at different granularities are complementary.

Mean Average Precision (MAP) improvement as the number of combined granularity scores increases.

As shown in the figure above, combining multiple granularity levels leads to a consistent increase in search accuracy. By integrating 20 different sets of acoustic patterns, the system reached a MAP of 26.32% using a simple averaging heuristic, more than doubling the performance of the baseline. Furthermore, the offline calculation of HMM training and KL-divergence ensures that the online matching phase remains highly efficient.

Conclusion

This multi-level acoustic pattern approach provides a robust and efficient framework for zero-resource Spoken Term Detection. By leveraging the complementary nature of different model granularities, the system significantly outperforms traditional unsupervised methods. Future work will focus on scaling this approach to larger, noisier "found data" and exploring automated techniques for optimizing granularity weights across different languages.

Summary updated

BibTeX
@inproceedings{chung2014unsupervised,
  author = {Cheng-Tao Chung and Chun-an Chan and Lin-Shan Lee},
  title = {{Unsupervised Spoken Term Detection with Spoken Queries by Multi-level Acoustic Patterns with Varying Model Granularity}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2014},
  doi = {10.1109/icassp.2014.6855121},
}
Figure