Visually Grounded Models of Spoken Language: A Survey of Datasets, Architectures and Evaluation Techniques

Grzegorz Chrupala

Introduction

Visually Grounded Models of Spoken Language (VGSL) represent a paradigm shift in artificial intelligence, moving away from text-centric processing toward a multi-modal approach that mimics human language acquisition. Unlike traditional Natural Language Processing (NLP) which relies on written text, or Automatic Speech Recognition (ASR) which focuses on transcription, VGSL models attempt to learn the structure and meaning of language by "grounding" raw audio in visual contexts, such as images or video. This survey provides a roadmap of the field's evolution, from early modular simulations to modern end-to-end deep learning architectures.

Methodology and Technical Evolution

The field has progressed through two distinct phases. The "First Wave" (1999–2005) utilized small-scale datasets and hand-coded components to simulate infant language learning. The "Second Wave" (2015–Present) is defined by deep learning and end-to-end neural networks.

Modern architectures typically employ:

  • Audio Encoders: Speech spectrograms are processed as 2D images using Convolutional Neural Networks (CNNs) or as sequences via Recurrent Neural Networks (RNNs) and Transformers.
  • Embedding Alignment: Models often use a triplet loss function to project audio clips and their corresponding visual signals into a shared vector space. This mathematical proximity allows the model to associate sounds (e.g., the spoken word "dog") with visual concepts.
  • Interpretability: Researchers use diagnostic classifiers and Representational Similarity Analysis (RSA) to probe these "black box" models, discovering that they often mirror human linguistic hierarchies, capturing phonology in lower layers and semantics in higher ones.

Datasets

The rapid advancement of the "Second Wave" was facilitated by the release of several large-scale multi-modal datasets. These resources vary in scale, language, and the nature of the speech (e.g., read aloud vs. spontaneous).

Comparison of major datasets used in visually grounded speech research.

As shown in the table above, datasets like SpokenCOCO and Places Audio Captions provide the high-volume data necessary for deep learning, while Flickr Audio Captions remains a foundational benchmark. Newer datasets are increasingly focusing on "ecological validity" by incorporating spontaneous speech and video-based temporal grounding.

Key Findings and Results

Research in VGSL has yielded several significant insights:

  • Hierarchical Emergence: Models trained only on audio-visual pairs spontaneously develop internal representations of phonemes and words.
  • Architectural Bias: Even untrained, randomly initialized CNNs show an innate ability to segment speech, suggesting that specific neural architectures provide a strong inductive bias for language processing.
  • Cross-Modal Translation: Visual signals can act as a "pivot," allowing models to translate between languages (e.g., English to Hindi) without the need for bilingual text corpora.
  • Performance Gains: Retrieval metrics have improved dramatically; for instance, Recall@10 on the Flickr dataset increased from 0.179 in 2015 to over 0.402 by 2021.

Future Directions and Limitations

Despite these successes, challenges remain. Most current models rely on "read" captions, which lack the complexity of the messy, spontaneous speech encountered in real-world environments. Furthermore, while models excel at concrete visual concepts, they struggle with abstract language (e.g., "justice"). Future work is expected to focus on combining visual grounding with self-supervised learning techniques to bridge the gap between sensory perception and abstract reasoning.

Summary updated

Abstract

This survey provides an overview of the evolution of visually grounded models of spoken language over the last 20 years. Such models are inspired by the observation that when children pick up a language, they rely on a wide range of indirect and noisy clues, crucially including signals from the visual modality co-occurring with spoken utterances. Several fields have made important contributions to this approach to modeling or mimicking the process of learning language: Machine Learning, Natural Language and Speech Processing, Computer Vision and Cognitive Science. The current paper brings together these contributions in order to provide a useful introduction and overview for practitioners in all these areas. We discuss the central research questions addressed, the timeline of developments, and the datasets which enabled much of this work. We then summarize the main modeling architectures and offer an exhaustive overview of the evaluation metrics and analysis techniques.
BibTeX
@article{chrupala2022visually,
  author = {Grzegorz Chrupala},
  title = {{Visually Grounded Models of Spoken Language: A Survey of Datasets, Architectures and Evaluation Techniques}},
  journal = {J. Artif. Intell. Res.},
  year = {2022},
  doi = {10.1613/jair.1.12967},
}
Figure