Abstract
This survey provides an overview of the evolution of visually grounded models of spoken language over the last 20 years. Such models are inspired by the observation that when children pick up a language, they rely on a wide range of indirect and noisy clues, crucially including signals from the visual modality co-occurring with spoken utterances. Several fields have made important contributions to this approach to modeling or mimicking the process of learning language: Machine Learning, Natural Language and Speech Processing, Computer Vision and Cognitive Science. The current paper brings together these contributions in order to provide a useful introduction and overview for practitioners in all these areas. We discuss the central research questions addressed, the timeline of developments, and the datasets which enabled much of this work. We then summarize the main modeling architectures and offer an exhaustive overview of the evaluation metrics and analysis techniques.
Introduction: Learning Language Without Text
Traditional AI approaches to language are often siloed: Speech Processing focuses on transcribing audio to text, while Natural Language Processing (NLP) focuses almost exclusively on written text. However, humans don't learn language from text; we learn from multi-modal sensory input.
The core challenge addressed in this survey is computational language acquisition: How can a machine learn the structure and meaning of spoken language using only raw audio signals paired with visual scenes? This requires solving several sub-problems simultaneously:
- Segmentation: Breaking a continuous stream of audio into words or phonemes.
- Grounding: Mapping those audio segments to visual concepts (e.g., the sound "dog" to the image of a dog).
- Representation: Learning internal features that capture both the "form" (sound) and "meaning" (visual reference) of language.
The Evolution of Grounded Models
The history of the field is divided into two distinct "waves" of development.
The First Wave (1999–2005)
Early models, such as CELL (Cross-channel Early Lexical Learning), utilized modular, often hard-coded components. These systems relied on small, artificial datasets and basic statistical methods like Mutual Information to identify recurring associations between audio segments and simple visual features, such as shape histograms.
The Second Wave (2015–Present)
The current era is defined by End-to-End Deep Learning. Key architectural components now include:
- Visual Encoders: Pre-trained Convolutional Neural Networks (CNNs) like VGG or ResNet that extract high-level features from images.
- Audio Encoders: These treat speech spectrograms as 2D images (using CNNs) or as sequences (using Recurrent Neural Networks like GRUs).
- Embedding Alignment: The "secret sauce" is often a triplet loss function. This forces the model to move the vector representation of a spoken caption closer to its matching image in a shared "embedding space," while pushing it away from mismatched images.
- Matchmaps: Advanced models use "matchmaps" to localize specific words to specific regions of an image, allowing for fine-grained spatial and temporal alignment.
The Role of Datasets
The shift to deep learning was largely driven by the release of large-scale datasets that provided the necessary volume of data for training complex neural architectures. These datasets vary significantly in language, scale, and the type of speech recorded.
Key datasets include:
- Flickr Audio Captions: The catalyst for the second wave, featuring 40,000 spoken recordings of Flickr8k image captions.
- SpokenCOCO: A massive expansion based on MS-COCO, containing over 600,000 read-aloud captions.
- Places Audio Captions: A crucial dataset featuring spontaneous speech (rather than read-aloud text) in both English and Hindi, providing higher "ecological validity."
- Video Datasets: Newer benchmarks like Howto100m and Spoken Moments in Time move the field toward dynamic, temporal grounding.
Key Insights and Emergent Properties
Research in this area has yielded several striking findings regarding how models process language:
- Emergent Phonology: Models trained only to match images and audio implicitly learn to recognize phonemes. They develop internal "detectors" for speech sounds without ever being shown a transcript.
- Hierarchical Learning: Much like the human brain, neural networks show a hierarchy. Lower layers of the audio encoder focus on "form" (phonemes and signal features), while higher layers focus on "meaning" (lexical and semantic concepts).
- The Power of Multilinguality: Visual grounding can act as an "interlingua." Models trained on both English and Hindi audio paired with the same images can perform cross-lingual retrieval without a translation dictionary.
- Unit Discovery: By using techniques like Vector Quantization (VQ), researchers can force models to "discretize" audio into a finite set of symbols, effectively discovering a "vocabulary" of speech units from scratch.
Performance and Evaluation
Progress is typically measured through Intrinsic Evaluation (retrieval tasks). On the Flickr Audio dataset, the ability to correctly find a matching image in the top 10 results (Recall@10) improved from a score of 0.179 in 2015 to over 0.400 by 2021. Furthermore, models using spatial-temporal localization (MISA) significantly outperform those that pool the entire audio signal into a single vector, particularly on complex datasets like Places.
Challenges and Future Directions
Despite significant progress, several hurdles remain:
- Generalization: Most models are still "overfit" to the specific style of image captions and struggle with different types of speech or visual environments.
- Ecological Validity: Real-world language learning is messy. Unlike perfectly descriptive captions, a parent might say "Look at that!" while pointing at something not in the child's direct line of sight.
- The "Text" Crutch: While these models don't use speech transcripts, they often use visual encoders pre-trained on ImageNet, which was labeled using text. Truly "text-free" learning remains a hurdle.
- Future Directions: Future research is moving toward dialogue grounding (learning through interaction) and combining visual grounding with self-supervised audio pre-training to better capture the nuances of unwritten languages.
Summary updated