Towards Bilingual Lexicon Discovery From Visually Grounded Speech Audio

Emmanuel Azuh, David Harwath, James R. GlassInterspeech 2019 · DOI

Introduction

Traditional machine translation (MT) and speech-to-speech systems are notoriously data-hungry, typically relying on a "cascade" of complex models. This standard pipeline involves Automatic Speech Recognition (ASR) to transcribe speech into text, Machine Translation to convert the text between languages, and Text-to-Speech (TTS) to generate the final audio output.

However, this approach encounters a significant "text bottleneck." Collecting the massive amounts of transcribed and translated text required for training is prohibitively expensive for the majority of the world’s 7,000+ languages. Furthermore, many of these languages lack a formal written orthography entirely. This research explores a novel alternative: can we bypass text altogether and learn to align languages by grounding them in the shared physical world through images?

Methodology

The proposed system utilizes a three-stage pipeline to discover a bilingual lexicon from scratch using only spoken audio and corresponding images.

The three-stage pipeline for bilingual lexicon discovery: 1) Multimodal embedding, 2) Peak detection for pseudo-word discovery, and 3) Clustering and cross-lingual linking.

1. Multimodal Embedding

The researchers employ a dual-branch neural network architecture. A VGG16 model processes images, while a DAVEnet (Deep Audio-Visual Embedding network) processes spoken captions. These models map both modalities into a shared 1024-dimensional embedding space. The training objective ensures that semantically related inputs—such as an image of a "forest" and the spoken word "forest" in English or "जंगल" (jungle) in Hindi—are projected to nearby coordinates in this space.

2. Pseudo-Word Discovery

To identify discrete words within a continuous audio stream without transcripts, the authors use a technique inspired by "interval piling." By comparing an utterance to its nearest neighbors in the embedding space, the system generates a similarity curve. Locations where multiple neighbors exhibit high similarity are identified as "peaks," which likely represent distinct semantic units or "pseudo-words."

Visualization of cross-lingual similarity heatmaps and the resulting similarity curve used to identify "peaks" or pseudo-word boundaries.

3. Clustering and Linking

Once peaks are identified, they are grouped into categories:

  • Clustering: A Dirichlet Process Gaussian Mixture Model (DPGMM) clusters these peaks to group similar acoustic segments (e.g., all instances of the sound "dog") into pseudo-word categories within each language.
  • Linking: The system then constructs a graph where English and Hindi clusters serve as nodes. Edges are established based on visual embedding similarity. Finally, the Louvain community detection algorithm identifies "meta-clusters"—groups of audio segments from both languages that refer to the same visual concept.

Datasets

The study utilizes the English-Hindi Places Audio dataset, containing 84,480 triplets: an image (from the Places 205 dataset), an English spoken caption, and a Hindi spoken caption. This dataset provides the essential "visual bridge," allowing the model to recognize that disparate sounds in different languages refer to the same physical objects.

Key Insights

  • Vision as an Interlingua: Images serve as a universal pivot. If two speakers are looking at the same mountain, their respective descriptions can be aligned without a Rosetta Stone of text.
  • Unsupervised Segmentation: The model "discovers" the concept of a word as a byproduct of the embedding process, identifying recurring patterns that correlate with visual features.
  • Semantic Alignment: Unlike acoustic-based speech processing, this method prioritizes meaning. An English cluster for "water" and a Hindi cluster for "पानी" (paanee) are linked because they both gravitate toward similar visual regions (e.g., blue, wavy textures), despite their acoustic differences.

Results and Evaluation

The clusters were evaluated using "purity"—the percentage of segments in a cluster that contain the same word, verified via ASR transcripts for ground truth.

  • Lexicon Quality: The system successfully discovered 101 bilingual meta-clusters. For instance, the English cluster for "people" achieved a purity of 0.84, while its linked Hindi counterpart ("लोग") reached 0.56.
  • Bilingual Picture Dictionary: Qualitatively, the model could take a specific region of an image (e.g., a guitar) and correctly associate it with the corresponding audio segments in both languages.
  • Retrieval Success: The model demonstrated strong performance in cross-lingual retrieval tasks, such as using an English caption to find a semantically equivalent Hindi caption.

Limitations and Future Work

While promising, the method currently excels primarily at concrete nouns with clear visual signatures (e.g., "lighthouse," "water"). It struggles with abstract concepts, verbs, or function words like "the" or "is" that lack distinct visual representations. Additionally, the current implementation relies on parallel images (both languages describing the same image). Future research will investigate whether this can scale to non-parallel datasets containing similar objects and expand the lexicon beyond the current 100-concept limit toward a full-scale speech-to-speech translation system.

Summary updated

BibTeX
@inproceedings{azuh2019towards,
  author = {Emmanuel Azuh and David Harwath and James R. Glass},
  title = {{Towards Bilingual Lexicon Discovery From Visually Grounded Speech Audio}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-1718},
}
Figure