On the difficulty of a distributional semantics of spoken language

Grzegorz Chrupala, Lieke Gelderloos, Ákos Kádár, Afra AlishahiarXiv 2018 · OpenReview

Distributional semantics—the principle that words occurring in similar contexts share similar meanings—has revolutionized Natural Language Processing (NLP). Models like Word2vec and Skip-thought leverage this to learn powerful representations from raw text. However, applying these same principles to raw audio signals remains a significant challenge. This paper investigates why the "magic" of distributional semantics is so much harder to achieve with speech than with text.

The Problem: Acoustic Confounds

In text, the context of a word is almost purely semantic. In speech, however, the signal is "noisy," carrying information about the speaker’s identity, emotional state, background noise, and recording equipment. A model might conclude that two audio clips are similar simply because they share the same speaker or recording environment, completely ignoring the semantic content. The authors aim to determine if these acoustic confounds can be stripped away to reveal an underlying semantic signal.

Methodology: Audio2vec vs. SegMatch

The researchers compared two unsupervised approaches using a shared encoder architecture (1D-Convolutional layers followed by Gated Recurrent Units and self-attention):

  1. Audio2vec: Modeled after the text-based Skip-gram, this approach predicts audio segments surrounding a target segment. Two variants were tested: Audio2vec-C (predicts the next frame given the previous) and Audio2vec-U (predicts surrounding frames without sequential access).
  2. SegMatch: This novel contrastive approach trains the model to recognize that different parts of the same utterance belong together. By splitting an utterance and erasing a middle chunk (to prevent simple acoustic "stitching"), the model learns to make representations of the beginning and end segments similar to each other while remaining distinct from other utterances.

The Synthetic Proof of Concept

To isolate the impact of speaker variability, the authors used Synthetically spoken COCO, a dataset of 300,000 image captions generated via a uniform Text-to-Speech (TTS) API. By "turning off" speaker diversity, they created a clean environment to test if the models could capture meaning.

Table 1: Performance metrics on synthetic speech (COCO TTS), showing SegMatch's strong semantic correlation (RSA).

In this controlled setting, SegMatch performed impressively. As shown in the results above, it achieved a 10% Recall@10 on paraphrase retrieval and a 0.5 correlation with image-space similarities (RSA). This suggests that in the absence of speaker noise, the distributional signal in audio is indeed sufficient to learn semantic representations.

The Human Speech Challenge

When moving to natural human speech (using the Flickr8k and Places Audio Caption Corpora), the results shifted dramatically. Human speech introduces the "Speaker Identity Trap": the easiest way for a model to match segments is to identify the speaker's voice rather than the message's meaning.

Table 2: Performance metrics on human speech (Flickr8k), where semantic correlation (RSA) drops to zero.

As seen in the human speech results, while SegMatch maintained a superficially decent 12% Recall@10, its semantic correlation (RSA) plummeted to 0.0. This indicates that the model was successfully matching segments based on acoustic artifacts or speaker identity rather than semantic content. The researchers even discovered that in the Flickr8k dataset, consecutively numbered images often shared recording timestamps, creating a spurious acoustic signal that the model exploited.

Conclusion

The study concludes that natural speech variability represents an "intractable confounding factor" for current unsupervised models. While SegMatch proves that a semantic signal exists within the distributional structure of audio, extracting it from human speech requires overcoming the dominant acoustic signals of the speaker and environment. Future work may need to incorporate multi-modal grounding—using images or video as an "anchor"—to help models separate the message from the messenger.

Summary updated

Abstract

In the domain of unsupervised learning most work on speech has focused on discovering low-level constructs such as phoneme inventories or word-like units. In contrast, for written language, where there is a large body of work on unsupervised induction of semantic representations of words, whole sentences and longer texts. In this study we examine the challenges of adapting these approaches from written to spoken language. We conjecture that unsupervised learning of the semantics of spoken language becomes feasible if we abstract from the surface variability. We simulate this setting with a dataset of utterances spoken by a realistic but uniform synthetic voice. We evaluate two simple unsupervised models which, to varying degrees of success, learn semantic representations of speech fragments. Finally we present inconclusive results on human speech, and discuss the challenges inherent in learning distributional semantic representations on unrestricted natural spoken language.
BibTeX
@misc{chrupala2018difficulty,
  author = {Grzegorz Chrupala and Lieke Gelderloos and Ákos Kádár and Afra Alishahi},
  title = {{On the difficulty of a distributional semantics of spoken language}},
  year = {2018},
  eprint = {1803.08869},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1803.08869},
}
Figure