TL;DR
This paper introduces a novel approach to improve end-to-end "acoustics-to-word" (A2W) speech recognition systems by leveraging acoustically grounded word embeddings. By initializing or regularizing the A2W model's output layer with these pre-trained embeddings, the system achieves better performance, especially with limited training data and for handling out-of-vocabulary words.
What Problem Does This Paper Address?
Traditional automatic speech recognition (ASR) systems are often modular, involving separate acoustic, pronunciation, and language models. End-to-end ASR aims to simplify this by jointly optimizing a single model. "Acoustics-to-word" (A2W) systems are a type of end-to-end ASR that directly map acoustic signals to words. While simpler and more efficient for decoding than sub-word systems (like character-based models), A2W systems face challenges: they require vast amounts of training data to perform well, and they struggle with words not present in their training vocabulary (out-of-vocabulary or OOV words). This paper tackles these limitations by proposing methods to enhance A2W performance, particularly when data is scarce and when dealing with unseen words.
Methods
The core innovation lies in using "acoustically grounded word embeddings." These are vector representations of words that are learned to capture their acoustic-phonetic properties, rather than just semantic meaning. The authors adapt a multi-view learning approach, similar to prior work [12], to jointly learn two embedding functions: one that maps acoustic signals to a vector (acoustic view) and another that maps character sequences to a vector (character view). This joint learning encourages the embeddings to capture shared acoustic information.
These pre-trained acoustically grounded word embeddings are then integrated into the A2W model in two primary ways:
- Initialization: The final pre-softmax weight matrix of the A2W recognizer, which essentially acts as word embeddings, is initialized with these pre-trained embeddings.
- Regularization: During A2W training, the model's weights are encouraged to stay close to the pre-trained embeddings using a regularization term. This helps the model retain the discriminative acoustic information learned during pre-training.
- Freezing: As an extreme form of regularization, the prediction layer weights are frozen after initialization with the pre-trained embeddings. This approach also facilitates handling OOV words.
For OOV prediction, the "Frozen" model's ability to embed arbitrary character sequences is leveraged. When an <UNK> (unknown) token is predicted, the system rescores potential OOV words by generating their embeddings and including them in the decoding process.
Datasets
The primary dataset used is the Switchboard corpus, a standard benchmark for conversational telephone speech. Experiments are conducted on subsets of this corpus, using vocabularies of 4K, 10K, and 20K words, corresponding to minimum occurrence thresholds of 25, 6, and 2, respectively. For OOV rescoring experiments, a larger 34K vocabulary is used.
Key Insights and Ideas
- Acoustic vs. Semantic Embeddings: The paper demonstrates that embeddings capturing acoustic-phonetic similarity are more beneficial for A2W recognition than traditional semantic word embeddings. This is because A2W models directly deal with acoustic signals, making acoustic similarity crucial.
- Pre-training Matters: Pre-training the A2W model's output layer with acoustically grounded embeddings significantly improves performance, especially compared to random initialization or initialization with phone models.
- Regularization as a Bridge: Regularizing the A2W model's weights towards pre-trained embeddings helps the model retain valuable acoustic information and generalize better, particularly for rare words.
- OOV Handling with "Frozen" Embeddings: Freezing the prediction layer weights after initialization with acoustically grounded embeddings provides a surprisingly effective way to handle OOV words by allowing for rescoring with an expanded vocabulary without additional training.
Key Results
- Improved Word Error Rate (WER):
- Initializing the A2W model with acoustically grounded embeddings (Acoustic view init + AWES) reduced the WER by 0.8% on the SWB development set for a 10K vocabulary compared to a phone CTC initialization.
- Regularizing the A2W model towards these embeddings yielded further improvements, achieving a 13.7% WER on the 20K vocabulary SWB/CH test sets.
- The "Frozen" model with OOV rescoring achieved an absolute WER reduction of 1.4% on Switchboard for the 4K vocabulary compared to the baseline.
- Better Word Discrimination: Acoustically grounded word embeddings (AGWE) achieved significantly higher average precision (AP) in cross-view word discrimination tasks compared to embeddings derived from the A2W prediction layer weights after CTC training, indicating they capture more discriminative acoustic information.
Limitations and Future Work
- Data Efficiency: While the proposed methods improve performance with limited data, A2W systems still generally require more data than sub-word systems to reach peak performance.
- OOV Rescoring Complexity: The OOV rescoring mechanism, while effective, adds a computational step during decoding.
- Future Directions: The authors suggest exploring other types of embedding models and training criteria, as well as tighter integration of embedding and recognizer training. They also highlight the potential for applying these techniques to low-resource languages, given the encouraging results with small vocabulary sizes.
Summary updated