I See What You Hear: A Vision-Inspired Method to Localize Words

Mohammad Samragh, Arnav Kundu, Ting-Yao Hu, Aman Chadha, Ashish Srivastava, Minsik Cho, Oncel Tuzel, Devang Naik

TL;DR

This paper introduces a novel, vision-inspired approach to word localization in speech, treating audio as a 1D image. Their method significantly reduces model size and improves accuracy compared to prior work, enabling efficient and precise identification of specific words within audio streams.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems have become ubiquitous, but they can be computationally intensive. For many applications, like wake-word detection ("Hey Google," "Alexa"), a full ASR system isn't necessary. Instead, limited-vocabulary models are sufficient. These models need to not only detect if a specific word is spoken but also precisely pinpoint when it occurs in the audio stream. Accurate localization is crucial for privacy (only activating when intended) and efficiency (avoiding unnecessary processing). While keyword spotting has been studied, efficient and accurate localization of these keywords, especially in streaming audio, remains a challenge.

Methods

The core innovation lies in adapting object detection techniques from computer vision, specifically inspired by the YOLO (You Only Look Once) framework, to the domain of audio. The proposed model takes raw audio as input and processes it through a CNN backbone to extract features. These features are then fed into modules designed for:

  • Event Detection: Predicting the probability of a word's presence within a given audio segment. This uses a sigmoid activation and binary cross-entropy loss, with a novel "don't care" region to handle partial overlaps.
  • Localization: Predicting the start and end times (offset and length) of detected words. This draws inspiration from the CenterNet object detection approach, predicting offsets and lengths relative to segment centers.
  • Classification: A final classifier is introduced to address potential confusion between similar-sounding words and to handle cases where multiple words might be present in a single audio segment. This classifier uses a softmax function and cross-entropy loss, leveraging the detection scores to focus on relevant predictions.

The overall training objective is a combined loss function that optimizes all these tasks simultaneously. For inference, the system uses Non-Maximum Suppression (NMS), a standard computer vision technique, to refine the proposed word events.

Datasets

The paper primarily uses the LibriSpeech dataset for training and evaluation. They extract word start and end times using the Montreal Forced Aligner. The lexicon for their experiments consists of the 1000 most frequent words in the training set.

Key Insights and Ideas

  • Audio as a 1D Image: The fundamental insight is treating audio segments as 1D images, allowing the direct application of powerful visual object detection paradigms.
  • Unified Detection and Localization: Unlike previous methods that might separate detection and localization, this approach trains them jointly, leading to better performance.
  • Vision-Inspired Classifier for Robustness: The addition of a classifier module significantly improves robustness against word confusion and overlapping speech, which are common challenges.
  • Efficiency through Model Design: The architecture is designed to be lightweight and capable of processing streaming audio, making it suitable for real-time applications on resource-constrained devices.

Key Results

  • Model Size Reduction: The proposed model is 94% smaller than the SpeechYolo baseline.
  • Improved Accuracy: Achieves a 6.5% better F1 score compared to SpeechYolo on the LibriSpeech dataset for localizing 1000 words.
  • Streaming Capability: Unlike SpeechYolo's 1-second segment limitation, this model can process arbitrary-length audio streams.
  • Keyword Spotting Performance: Outperforms SpeechYolo in keyword spotting tasks, achieving better Term Weight Value (TWV) scores. For example, on the test_other partition, their large model achieves a TWV of 0.64 compared to SpeechYolo's 0.38.
  • Ablation Study: Demonstrates that each component (detection signal ŷ, classification result ŝ, and localization predictions ô, Î) contributes to overall performance, with ŷ improving recall, ŝ improving precision, and ô, Î improving localization accuracy (IOU).

Limitations and Future Work

  • Reproducibility: The authors note that they were unable to reproduce the exact results of SpeechYolo, as the authors did not provide pre-processed inputs.
  • Lexicon Size: While the model scales to 1000 words, future work could explore even larger vocabularies or open-vocabulary scenarios.
  • Real-World Noise: The current evaluation is primarily on clean datasets. Testing and improving performance in noisy, real-world environments would be a valuable next step.
  • End-to-End Training: While the model is trained end-to-end for detection and localization, further exploration of end-to-end training for the entire ASR pipeline could be beneficial.

Summary updated

Abstract

This paper explores the possibility of using visual object detection techniques for word localization in speech data. Object detection has been thoroughly studied in the contemporary literature for visual data. Noting that an audio can be interpreted as a 1-dimensional image, object localization techniques can be fundamentally useful for word localization. Building upon this idea, we propose a lightweight solution for word detection and localization. We use bounding box regression for word localization, which enables our model to detect the occurrence, offset, and duration of keywords in a given audio stream. We experiment with LibriSpeech and train a model to localize 1000 words. Compared to existing work [1], our method reduces model size by 94%, and improves the F1 score by 6.5%.
BibTeX
@inproceedings{samragh2022i,
  author = {Mohammad Samragh and Arnav Kundu and Ting-Yao Hu and Aman Chadha and Ashish Srivastava and Minsik Cho and Oncel Tuzel and Devang Naik},
  title = {{I See What You Hear: A Vision-Inspired Method to Localize Words}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
}
Figure