Domain-Informed Probing of wav2vec 2.0 Embeddings for Phonetic Features

Patrick Cormac English, John D. Kelleher, Julie Carson-BerndsenNAACL 2022

TL;DR

This paper investigates how well the wav2vec 2.0 model, a powerful self-supervised speech representation learner, encodes phonetic information across its different layers. By using a "probing" technique, they found that wav2vec 2.0 progressively learns to distinguish phonetic features, with later layers showing a more sophisticated understanding of phoneme relationships, aligning with linguistic expectations.

What Problem Does This Paper Address?

Transformer models, like wav2vec 2.0, have revolutionized speech processing by learning rich representations from vast amounts of unlabeled audio data. While their performance on downstream tasks like speech recognition is impressive, there's a significant gap in understanding what linguistic information is actually encoded within their internal representations, especially at the phonetic level. This paper aims to bridge that gap by specifically probing the wav2vec 2.0 model to see how and when phonetic features emerge and are represented across its transformer layers.

Methods

The core methodology is probing, a technique adapted from natural language processing. Here's how it works:

  1. Embedding Extraction: The wav2vec 2.0 "base_960" model is used to generate embeddings for each layer of its transformer component for audio segments from the TIMIT dataset.
  2. Phoneme Averaging: For each phoneme in the TIMIT dataset, the embeddings from the corresponding audio frames are averaged. This creates a single "phoneme-averaged" representation for each phoneme at each layer.
  3. Probing Task: A simple Multi-Layer Perceptron (MLP) is trained for each layer. The input to the MLP is the phoneme-averaged embedding, and the target output is the corresponding TIMIT phoneme label.
  4. Evaluation: The accuracy of these MLPs in predicting phoneme labels, as well as features like manner of articulation (MOA) and place of articulation (POA), is measured for each layer.
  5. Hierarchical Clustering: To visualize how the model groups similar phonemes, dendrograms are generated based on confusion matrices from the probing task, focusing on obstruent sounds.

The novelty lies in applying this established probing methodology to the specific context of wav2vec 2.0's layered transformer architecture and focusing on phonetic features.

Datasets

  • TIMIT: This corpus of American-accented English speech was used. It's valuable because it provides high-quality, time-aligned phonetic transcriptions, which are crucial for the probing task. The dataset consists of 5.4 hours of audio.
  • LibriSpeech: This large, unlabeled dataset was used for pre-training the wav2vec 2.0 model itself. The paper focuses on the pre-trained "base_960" model.

Key Insights and Ideas

  • Layer-wise Phonetic Learning: The model doesn't learn all phonetic information at once. Instead, it progressively refines its understanding of phonetic features as the audio signal passes through its transformer layers.
  • Emergence of Articulation Features: The accuracy for predicting manner and place of articulation features shows a significant increase around layer 7, suggesting that these more abstract phonetic properties are learned in the mid-to-later layers.
  • Domain Alignment: The hierarchical clustering of phonemes in later layers shows groupings that align well with linguistic expectations (e.g., stops and closures becoming proximate). This indicates the model is learning representations that mirror human linguistic understanding.
  • Distinguishing Difficult Phonemes: The paper suggests that the contextual information from the transformer helps in distinguishing acoustically similar phonetic details that are otherwise hard to separate.

Key Results

  • Categorization Accuracy: Accuracy for predicting phoneme labels, MOA, and POA generally increases with layer depth, with notable improvements around layer 7.
  • MOA and POA Accuracy: Robust results for MOA and POA categorization are achieved around layer 7, indicating this is a key stage for learning these articulatory features.
  • Hierarchical Clustering: Dendrograms show a clear evolution in how phonemes are clustered. In early layers, clusters are less refined. By layer 7 and 12, clusters for related phonemes (e.g., stops and their closures, or fricatives with similar places of articulation) become more coherent and align with linguistic categories. For instance, the distinction between labio-dental and dental fricatives becomes clearer in later layers.

Limitations and Future Work

  • Label Imbalance: The authors acknowledge that label imbalance in the TIMIT dataset was not specifically accounted for in the probing task and plan to address this in future work.
  • Chance Correlation: While a control experiment was performed to rule out chance correlation, the authors note that label imbalance could still influence results.
  • Mechanism of Change: The paper demonstrates that phonetic information emerges but doesn't fully explain how the transformer's attention mechanisms or feature extractor modules contribute to these layer-wise changes.
  • Future Directions: Future work will focus on:
  • More concretely establishing the nature and effective mechanisms of layer-wise phonetic learning.
  • Exploring the emergence of phonological generalizations.
  • Investigating the relationship of these emerging phonetic categories to infant language acquisition.
  • Further exploring the role of feature extractor modules and attention matrices.

Summary updated

BibTeX
@inproceedings{english2022domain,
  author = {Patrick Cormac English and John D. Kelleher and Julie Carson-Berndsen},
  title = {{Domain-Informed Probing of wav2vec 2.0 Embeddings for Phonetic Features}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure