Semantic Query-by-example Speech Search Using Visual Grounding

Herman Kamper, Aristotelis Anastassiou, Karen Livescu

TL;DR

This paper introduces a novel approach to semantic query-by-example (QbE) speech search that leverages visually grounded speech models trained on untranscribed audio-visual data. This method significantly outperforms traditional acoustic-only QbE systems, especially for semantic retrieval, and offers a promising solution for low-resource languages where text transcriptions are scarce.

What Problem Does This Paper Address?

Traditional speech processing systems often rely heavily on large amounts of transcribed data. However, many languages and domains lack such resources, hindering the development of effective speech technologies. This paper tackles the challenge of building robust speech search systems, specifically Query-by-Example (QbE), in low-resource settings. QbE involves finding spoken utterances that match a given spoken query. The authors go a step further by focusing on semantic QbE, aiming to retrieve utterances that are not just exact matches but also semantically related to the query's meaning. This is particularly difficult without transcriptions, as traditional methods often rely on phonetic or textual alignments.

Methods

The core of the paper's contribution lies in using visual grounding as a weak supervision signal to train an acoustic embedding function. Instead of relying on text transcriptions, the model learns to associate spoken words with concepts present in accompanying images.

Here's a breakdown of their approach:

  • Visually Grounded Acoustic Embedding: They employ a Convolutional Neural Network (CNN) that takes speech segments as input and outputs a fixed-dimensional vector. This network is trained using paired images and their unlabelled spoken captions. An external image tagger provides "soft textual labels" (probabilities of words appearing in the image) which serve as training targets for the speech network. This allows the model to learn embeddings that capture semantic meaning rather than just phonetic similarity.
  • Embedding-based QbE: The trained acoustic embedding function maps speech segments (queries and search utterances) into a shared vector space. QbE is then performed by comparing the embedding of the query to the embeddings of segments within the search database, typically using cosine distance.
  • Two QbE Strategies:
    • FAST: Each search utterance is embedded as a single vector. This is computationally efficient but might dilute the query's signal with other content in the utterance.
  • DENSE: Each search utterance is split into overlapping sub-segments, and each segment is embedded individually. The query embedding is then compared to all these sub-segment embeddings, and the minimum distance determines the utterance's relevance. This is more computationally intensive but can potentially yield better results and even localize the matching segment.

Datasets

  • Training Data: The authors utilize a corpus of 8000 images with 5 spoken captions each, totaling approximately 37 hours of speech. This dataset is described as having parallel images and unlabelled spoken captions.
  • Evaluation Data: For evaluation, they extract spoken queries corresponding to 67 keyword types. A disjoint part of the dataset is used as the search collection. For semantic evaluation, they collected semantic labels for 1000 test utterances using Amazon Mechanical Turk, annotating whether an utterance was semantically relevant to a query.

Key Insights and Ideas

  • Visual Grounding as a Powerful Weak Supervisor: The paper demonstrates that visual context, even without explicit text transcriptions, can provide sufficient semantic information to train effective speech embedding models for tasks like QbE. This is a significant step towards building speech systems for unwritten languages or domains with limited annotated data.
  • Semantic QbE Outperforms Exact QbE for Visually Grounded Models: While traditional QbE systems (like DTW) tend to perform worse on semantic retrieval compared to exact matching, the visually grounded models (FASTGROUNDED and DENSEGROUNDED) show improved performance on semantic QbE. This suggests that the learned visual semantics are more aligned with meaning-based retrieval.
  • DENSE Offers a Strong Trade-off: The DENSE approach, while slower than FAST, provides a significant performance boost for both exact and semantic QbE, outperforming even traditional DTW methods while being considerably faster.

Key Results

  • Superior Semantic Retrieval: The DENSEGROUNDED system achieved the best overall performance on semantic QbE metrics, particularly excelling in Spearman's ρ (correlation with human judgments), indicating a strong alignment with semantic relevance.
  • Competitive Exact Retrieval: Both FASTGROUNDED and DENSEGROUNDED systems significantly outperformed the baseline DTW for exact QbE, with DENSEGROUNDED showing particularly strong results.
  • Speed Advantage: DENSEGROUNDED was over 6 times faster than the DTW baseline. FASTGROUNDED was orders of magnitude faster but with a performance trade-off.
  • Supervised vs. Visually Grounded: While supervised systems (trained with text labels) generally performed best, the visually grounded systems achieved competitive results, especially considering the lack of transcriptions.

Limitations and Future Work

  • Localization Not Evaluated: The authors note that while the DENSE approach can predict the location of a matching segment, this specific performance aspect was not evaluated in the paper.
  • Hyper-parameter Optimization: Due to the computational cost of the DENSE systems, extensive hyper-parameter optimization was not feasible.
  • Intermediate Layer Embeddings: The paper mentions considering intermediate layers of the network for embeddings but focused on the final output layer for evaluation.
  • Future Directions: The authors plan to explore other vision+speech models for acoustic embeddings, investigate acoustic-only methods for semantic retrieval, and specifically apply their visual supervision techniques to truly low-resource languages. They also suggest exploring other normalization techniques for the FAST approach.

Summary updated

Abstract

A number of recent studies have started to investigate how speech systems can be trained on untranscribed speech by leveraging accompanying images at training time. Examples of tasks include keyword prediction and within- and across-mode retrieval. Here we consider how such models can be used for query-by-example (QbE) search, the task of retrieving utterances relevant to a given spoken query. We are particularly interested in semantic QbE, where the task is not only to retrieve utterances containing exact instances of the query, but also utterances whose meaning is relevant to the query. We follow a segmental QbE approach where variable-duration speech segments (queries, search utterances) are mapped to fixed-dimensional embedding vectors. We show that a QbE system using an embedding function trained on visually grounded speech data outperforms a purely acoustic QbE system in terms of both exact and semantic retrieval performance.
BibTeX
@inproceedings{kamper2019semantic,
  author = {Herman Kamper and Aristotelis Anastassiou and Karen Livescu},
  title = {{Semantic Query-by-example Speech Search Using Visual Grounding}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=ubuFlhZYpdb},
}
Figure