Acoustically Grounded Word Embeddings for Improved Acoustics-to-word Speech Recognition

Shane Settle, Kartik Audhkhasi, Karen Livescu, Michael Picheny

TL;DR

This paper introduces a novel approach to improve end-to-end "acoustics-to-word" (A2W) speech recognition systems by leveraging acoustically grounded word embeddings. By initializing or regularizing the A2W model's output layer with these pre-trained embeddings, the system achieves better performance, especially with limited training data and for handling out-of-vocabulary words.

What Problem Does This Paper Address?

Traditional automatic speech recognition (ASR) systems are often modular, involving separate acoustic, pronunciation, and language models. End-to-end ASR aims to simplify this by jointly optimizing a single model. "Acoustics-to-word" (A2W) systems are a type of end-to-end ASR that directly map acoustic signals to words. While simpler and more efficient for decoding than sub-word systems (like character-based models), A2W systems face challenges: they require vast amounts of training data to perform well, and they struggle with words not present in their training vocabulary (out-of-vocabulary or OOV words). This paper tackles these limitations by proposing methods to enhance A2W performance, particularly when data is scarce and when dealing with unseen words.

Methods

The core innovation lies in using "acoustically grounded word embeddings." These are vector representations of words that are learned to capture their acoustic-phonetic properties, rather than just semantic meaning. The authors adapt a multi-view learning approach, similar to prior work [12], to jointly learn two embedding functions: one that maps acoustic signals to a vector (acoustic view) and another that maps character sequences to a vector (character view). This joint learning encourages the embeddings to capture shared acoustic information.

These pre-trained acoustically grounded word embeddings are then integrated into the A2W model in two primary ways:

  1. Initialization: The final pre-softmax weight matrix of the A2W recognizer, which essentially acts as word embeddings, is initialized with these pre-trained embeddings.
  2. Regularization: During A2W training, the model's weights are encouraged to stay close to the pre-trained embeddings using a regularization term. This helps the model retain the discriminative acoustic information learned during pre-training.
  3. Freezing: As an extreme form of regularization, the prediction layer weights are frozen after initialization with the pre-trained embeddings. This approach also facilitates handling OOV words.

For OOV prediction, the "Frozen" model's ability to embed arbitrary character sequences is leveraged. When an <UNK> (unknown) token is predicted, the system rescores potential OOV words by generating their embeddings and including them in the decoding process.

Datasets

The primary dataset used is the Switchboard corpus, a standard benchmark for conversational telephone speech. Experiments are conducted on subsets of this corpus, using vocabularies of 4K, 10K, and 20K words, corresponding to minimum occurrence thresholds of 25, 6, and 2, respectively. For OOV rescoring experiments, a larger 34K vocabulary is used.

Key Insights and Ideas

  • Acoustic vs. Semantic Embeddings: The paper demonstrates that embeddings capturing acoustic-phonetic similarity are more beneficial for A2W recognition than traditional semantic word embeddings. This is because A2W models directly deal with acoustic signals, making acoustic similarity crucial.
  • Pre-training Matters: Pre-training the A2W model's output layer with acoustically grounded embeddings significantly improves performance, especially compared to random initialization or initialization with phone models.
  • Regularization as a Bridge: Regularizing the A2W model's weights towards pre-trained embeddings helps the model retain valuable acoustic information and generalize better, particularly for rare words.
  • OOV Handling with "Frozen" Embeddings: Freezing the prediction layer weights after initialization with acoustically grounded embeddings provides a surprisingly effective way to handle OOV words by allowing for rescoring with an expanded vocabulary without additional training.

Key Results

  • Improved Word Error Rate (WER):
    • Initializing the A2W model with acoustically grounded embeddings (Acoustic view init + AWES) reduced the WER by 0.8% on the SWB development set for a 10K vocabulary compared to a phone CTC initialization.
  • Regularizing the A2W model towards these embeddings yielded further improvements, achieving a 13.7% WER on the 20K vocabulary SWB/CH test sets.
  • The "Frozen" model with OOV rescoring achieved an absolute WER reduction of 1.4% on Switchboard for the 4K vocabulary compared to the baseline.
  • Better Word Discrimination: Acoustically grounded word embeddings (AGWE) achieved significantly higher average precision (AP) in cross-view word discrimination tasks compared to embeddings derived from the A2W prediction layer weights after CTC training, indicating they capture more discriminative acoustic information.

Limitations and Future Work

  • Data Efficiency: While the proposed methods improve performance with limited data, A2W systems still generally require more data than sub-word systems to reach peak performance.
  • OOV Rescoring Complexity: The OOV rescoring mechanism, while effective, adds a computational step during decoding.
  • Future Directions: The authors suggest exploring other types of embedding models and training criteria, as well as tighter integration of embedding and recognizer training. They also highlight the potential for applying these techniques to low-resource languages, given the encouraging results with small vocabulary sizes.

Summary updated

Abstract

Direct acoustics-to-word (A2W) systems for end-to-end automatic speech recognition are simpler to train, and more efficient to decode with, than sub-word systems. However, A2W systems can have difficulties at training time when data is limited, and at decoding time when recognizing words outside the training vocabulary. To address these shortcomings, we investigate the use of recently proposed acoustic and acoustically grounded word embedding techniques in A2W systems. The idea is based on treating the final pre-softmax weight matrix of an AWE recognizer as a matrix of word embedding vectors, and using an externally trained set of word embeddings to improve the quality of this matrix. In particular we introduce two ideas: (1) Enforcing similarity at training time between the external embeddings and the recognizer weights, and (2) using the word embeddings at test time for predicting out-of-vocabulary words. Our word embedding model is acoustically grounded, that is it is learned jointly with acoustic embeddings so as to encode the words' acoustic-phonetic content; and it is parametric, so that it can embed any arbitrary (potentially out-of-vocabulary) sequence of characters. We find that both techniques improve the performance of an A2W recognizer on conversational telephone speech.
BibTeX
@inproceedings{settle2019acoustically,
  author = {Shane Settle and Kartik Audhkhasi and Karen Livescu and Michael Picheny},
  title = {{Acoustically Grounded Word Embeddings for Improved Acoustics-to-word Speech Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=s6Gd-rWn1BF},
}
Figure