Abstract
This paper demonstrates three different systems capable of performing the multimodal word discovery task. A multimodal word discovery system accepts, as input, a database of spoken descriptions of images (or a set of corresponding phone transcripts), and learns a lexicon which is a mapping from phone strings to their associated image concepts. Three systems are demonstrated: one based on a statistical machine translation (SMT) model, and two based on neural machine translation (NMT). On the Flickr8k dataset, the SMT-based model performs significantly better than the NMT-based variants, achieving a 49.6% F1 score. Finally, we apply our word discovery system to the task of image retrieval and achieve 29.1% recall@10 on the standard 1000-image Flickr8k test set.
Introduction
For many of the world's languages, a digital lexicon or even a written form does not exist. This lack of resources makes building speech recognition tools prohibitively expensive. "Word discovery"—the process of segmenting a continuous stream of sounds into meaningful units—is a critical first step in addressing this challenge.
While traditional research often focuses on finding recurring patterns in raw audio, this study utilizes images as a source of weak supervision. When a speaker describes a photo, the objects present (e.g., "dog," "tree," "blue shirt") serve as a "bag of noisy labels" for the accompanying speech. The core challenge is to determine which specific sequence of phones (phonetic sounds) maps to which visual concept without any pre-existing dictionary.
Methodology
The authors frame the word discovery task as a machine translation problem, where a sequence of phones is "translated" into a set of image concepts. They evaluated three distinct architectures:
- Statistical Machine Translation (SMT): Based on classic IBM alignment models, this system uses the Expectation-Maximization (EM) algorithm to find the most likely translation pairs. It assumes each phone is generated by exactly one image concept (or a "NULL" concept).
- NMT (Normalized-over-time): A standard neural encoder-decoder with an attention mechanism that considers the entire phone sequence to predict concepts.
- NMT (Normalized-over-concepts): A modified neural approach where the attention mechanism is forced to be sparse. It requires each phone to choose a specific concept, mimicking the one-to-one nature of words more closely than standard attention.
Key Results and Performance
The study utilized the Flickr8k and Flickr30kEntities datasets, converting captions into phone sequences using the CMU pronunciation dictionary. The SMT model emerged as the clear winner in this low-data, unsupervised context.
As shown in the results table, the SMT model achieved a 49.6% F1-score, vastly outperforming the best NMT variant (31.0%). In the downstream task of image retrieval, the SMT system achieved 29.1% Recall@10, outperforming previous end-to-end speech-to-image retrieval baselines (17.9%).
Insights into Model Behavior
One of the most significant findings was the "Small Data" advantage of SMT. While neural models often dominate in high-resource settings, they struggled to converge on the ~100,000 phone dataset used here.
Length Bias and Segmentation
A qualitative analysis revealed distinct biases in how these models segment speech. NMT models tend to "over-segment" or merge multiple words into long pseudo-words. Conversely, SMT tends to produce very short words, often under three phones in length.
The "NULL" and Context Problems
A major source of error across all models is the "NULL" class—phones representing grammar or filler words (like "the" or "is") that lack visual counterparts. Additionally, the models currently treat images as a "bag of concepts," ignoring spatial relationships (e.g., a "hand" is usually near a "person") that could help resolve phonetic ambiguities.
Conclusion
This research highlights that traditional statistical methods remain highly effective for low-resource, multimodal word discovery. Future work will focus on incorporating richer context, such as actions (verbs) and poses, and integrating live image classifiers to learn directly from pixels rather than pre-defined text tags.
Summary updated