Semantic keyword spotting by learning from images and speech

Herman Kamper, Gregory Shakhnarovich, Karen Livescu

TL;DR

This paper introduces a novel approach to semantic speech retrieval by training a model to map untranscribed speech to visual scene descriptions. This visually grounded model can identify utterances semantically relevant to a text query, even without exact word matches, and outperforms transcription-based methods in capturing nuanced semantic relationships.

What Problem Does This Paper Address?

Traditional speech recognition and retrieval systems heavily rely on large amounts of transcribed speech data. This is a significant limitation for low-resource languages, endangered languages, or scenarios where obtaining transcriptions is impractical. The paper addresses this by exploring how to learn semantic understanding from speech when only paired visual context (images) and untranscribed audio are available. This is crucial for applications like robotics, where audio-visual cues are common, and for advancing language acquisition research. The core challenge is to move beyond simple keyword spotting (exact word matches) to a more flexible "semantic speech retrieval" where the system understands the meaning of the speech in relation to a query, even if the query words aren't explicitly present.

Methods

The authors propose a "visually grounded speech model" that leverages paired images and untranscribed speech. The key innovation lies in using an external image tagger to generate "soft text labels" (probabilities of words being present in an image) for each training image. These soft labels then serve as targets for a neural network that maps acoustic speech features to these semantic labels. This network, trained without any transcriptions, learns to predict which words are semantically associated with a given speech utterance.

For training, they use a convolutional neural network (CNN) architecture for processing speech features (MFCCs). The loss function is a summed cross-entropy loss, adapted for multi-label classification. At test time, the model takes only the speech input and outputs a score for each potential keyword, indicating its semantic relevance. This score is then used for semantic speech retrieval.

Datasets

The paper utilizes a subset of the corpus from [91], which consists of parallel images and spoken captions. For training their visual tagger, they combine data from Flickr30k and MSCOCO, ensuring no overlap with the image-speech data used for the speech model.

Crucially, the authors introduce a newly collected dataset for semantic speech retrieval. This dataset consists of 1000 spoken utterances from the aforementioned corpus, each annotated by multiple human annotators (Amazon Mechanical Turk) for semantic relevance to a set of 70 keywords. This dataset is valuable because it provides fine-grained, multi-annotator semantic relevance judgments, allowing for a more nuanced evaluation than simple binary labels.

Key Insights and Ideas

  • Visual context as a powerful semantic bridge: The research demonstrates that visual information, even in the form of soft image tags, can effectively ground speech semantics, enabling retrieval of semantically related utterances without relying on transcriptions.
  • Beyond verbatim matches: The visually grounded model excels at retrieving utterances that are semantically relevant but not exact matches to the query keyword, a capability that transcription-based models struggle with.
  • Soft labels for nuanced semantics: Using soft labels from image taggers and soft human judgments for evaluation allows for a more accurate assessment of semantic understanding, acknowledging the inherent ambiguity in semantic relationships.
  • Visual grounding complements, and sometimes surpasses, text supervision: While transcription-based models achieve high precision on exact matches, the visually grounded model demonstrates better performance in matching soft human judgments and capturing non-verbatim semantic relationships.

Key Results

  • The visually grounded model (VISIONSPEECHCNN) achieves a semantic speech retrieval precision of almost 60% on its top ten retrievals without seeing any text transcriptions during training.
  • Compared to a supervised model trained on transcriptions (SUPERVISEDBOWCNN), VISIONSPEECHCNN matches human judgments better by some measures, particularly in retrieving non-verbatim semantic matches.
  • In semantic speech retrieval, VISIONSPEECHCNN outperforms baseline models (TEXTPRIOR, VISIONTAGPRIOR) and even a "cheating" model that uses only visual tagger outputs (VISIONCNN) on several metrics, indicating it effectively leverages acoustic information.
  • The paper's analysis shows that VISIONSPEECHCNN makes more correct semantic (non-exact) predictions than exact predictions, highlighting its strength in nuanced semantic understanding.

Limitations and Future Work

  • Cascaded ASR limitations: While a cascaded approach (ASR + text retrieval) can perform well with perfect ASR, its performance degrades significantly with increasing word error rates, suggesting that visually grounded models offer a more robust alternative in low-resource or noisy ASR scenarios.
  • Human judgment ambiguity: The authors acknowledge that semantic relevance is inherently ambiguous, which is why they collected multi-annotator judgments and used metrics like Spearman's rho to account for this.
  • Future directions: The authors suggest exploring how to combine visual and textual supervision signals when both are available. They also propose investigating the model's ability to localize specific acoustic segments responsible for semantic concepts and exploring the cognitive implications of their model for language acquisition research. They also encourage the use of their newly released dataset.

Summary updated

Abstract

There is growing interest in models that can learn from unlabelled speech paired with visual context. This setting is relevant for low-resource speech processing, robotics, and human language acquisition research. Here we study how a visually grounded speech model, trained on images of scenes paired with spoken captions, captures aspects of semantics. We use an external image tagger to generate soft text labels from images, which serve as targets for a neural model that maps untranscribed speech to (semantic) keyword labels. We introduce a newly collected data set of human semantic relevance judgements and an associated task, semantic speech retrieval, where the goal is to search for spoken utterances that are semantically relevant to a given text query. Without seeing any text, the model trained on parallel speech and images achieves a precision of almost 60% on its top ten semantic retrievals. Compared to a supervised model trained on transcriptions, our model matches human judgements better by some measures, especially in retrieving non-verbatim semantic matches. We perform an extensive analysis of the model and its resulting representations.
BibTeX
@article{kamper2019semantic,
  author = {Herman Kamper and Gregory Shakhnarovich and Karen Livescu},
  title = {{Semantic keyword spotting by learning from images and speech}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2019},
  url = {https://openreview.net/forum?id=oy7uQanbC-i},
}
Figure