TL;DR
This paper introduces a novel approach to semantic speech retrieval by training a model on untranscribed speech paired with images. This visually grounded model can identify spoken utterances relevant to a text query, even if the exact words aren't present, achieving impressive results without relying on traditional speech transcriptions.
What Problem Does This Paper Address?
Traditional Automatic Speech Recognition (ASR) systems require vast amounts of transcribed speech data, making them impractical for low-resource scenarios. This paper tackles the challenge of learning semantic meaning from speech in a more flexible way, specifically by leveraging the rich context provided by paired images. The goal is to move beyond simply transcribing speech to understanding its meaning, enabling tasks like searching for spoken content based on semantic relevance rather than exact keyword matching. This is crucial for applications in areas like robotics, human language acquisition, and situations where speech data is scarce or untranscribed.
Methods
The core of the proposed method is a "visually grounded speech model." This model is trained using a two-stage process:
- Visual Tagging: An external image tagging system is used to generate "soft text labels" for training images. These labels represent the probability of various words being present in the image.
- Speech-to-Label Mapping: A neural network (specifically, a Convolutional Neural Network or CNN) is trained on untranscribed spoken captions. The soft text labels generated from the corresponding images serve as the targets for this network. The network learns to map acoustic features of speech to these semantic labels.
During testing, the trained speech model takes untranscribed speech as input and outputs a probability distribution over keywords, indicating how semantically relevant each keyword is to the utterance. This output is then used for semantic speech retrieval. The paper also introduces a new dataset and task: "semantic speech retrieval," where the objective is to find spoken utterances semantically related to a given text query, even if the query word isn't explicitly spoken.
Datasets
- Training Data: The model is trained on a corpus of parallel images and spoken captions from Harwath & Glass [6]. This corpus contains approximately 8000 images with five spoken captions each, totaling around 37 hours of active speech. The images used for training the visual tagger are distinct from those in the speech-image corpus.
- Evaluation Data: A new dataset of human semantic relevance judgments was collected for evaluation. This dataset was created by extending the corpus of [6] and using Amazon Mechanical Turk (AMT) to gather annotations from human annotators. Annotators indicated which keywords could be used to search for a given utterance describing a scene.
Key Insights and Ideas
- Visual Context is Powerful for Semantics: The paper strongly suggests that pairing speech with visual context allows models to capture semantic meaning more effectively than relying solely on text transcriptions, especially for non-verbatim matches.
- Soft Labels Bridge the Gap: Using "soft text labels" from image taggers as targets for the speech model is a key innovation. This approach avoids the need for exact transcriptions and allows the model to learn more nuanced semantic relationships.
- Semantic Retrieval Beyond Exact Matches: The visually grounded model excels at retrieving utterances that are semantically related but don't contain the exact query word. This is a significant advantage over traditional keyword spotting.
- Human Judgements are "Soft": The collection of human relevance judgments revealed a wide range of opinions, highlighting that semantic relevance is often subjective and "soft," which the proposed model is well-suited to capture.
Key Results
- Strong Semantic Retrieval Performance: The VISIONSPEECHCNN model, trained without transcriptions, achieved a precision of almost 60% on its top ten semantic retrievals (P@10).
- Outperforming Baselines: The visually grounded model significantly outperformed baseline models that relied on simple prior probabilities or image tagger outputs alone.
- Competitive with Supervised Models: While fully supervised models trained on transcriptions achieved higher precision on exact keyword spotting and some semantic metrics (like P@10), the VISIONSPEECHCNN model showed better performance on metrics that credit non-verbatim semantic matches and on Spearman's rho, which measures correlation with soft human ratings.
- Better Human Judgment Correlation: For Spearman's rho, the visually trained model outperformed all others, indicating its superior ability to align with subjective human judgments of semantic relevance.
Limitations and Future Work
- Limitations: The paper acknowledges that fully supervised models still outperform their visually grounded model on some metrics, particularly those based on hard ground truth labels. The performance of the visually grounded model is also dependent on the quality of the external image tagger.
- Future Work: The authors suggest that further exploration of the learned representations and their potential for other downstream tasks would be valuable. Investigating how to further improve the model's performance on hard ground truth metrics while retaining its strength in soft semantic matching is also a promising direction.
Summary updated