Representation Learning for Speech Recognition Using Feedback Based Relevance Weighting

Purvi Agrawal, Sriram GanapathyICASSP 2021 · DOI

TL;DR

This paper introduces a novel two-stage approach for learning speech representations by incorporating feedback from previous predictions into a relevance weighting mechanism. This method significantly improves Automatic Speech Recognition (ASR) performance on challenging datasets by adaptively selecting relevant acoustic and modulation features.

What Problem Does This Paper Address?

Learning effective representations from raw audio signals for Automatic Speech Recognition (ASR) remains a significant challenge. While traditional methods rely on hand-crafted features like mel filterbanks, recent deep learning approaches have explored learning filterbanks directly from raw waveforms. However, these methods often struggle with interpretability and achieving consistent performance gains, especially in noisy environments. This paper aims to bridge this gap by developing a more robust and adaptive representation learning framework that can better capture the nuances of speech signals, even under adverse conditions.

Methods

The proposed approach is a two-stage process:

  1. Acoustic Filterbank Learning: This stage begins with learning acoustic filterbanks from raw audio waveforms. A key innovation here is the introduction of a "relevance weighting" mechanism. This mechanism uses embeddings of the model's outputs from previous time steps as feedback to a relevance weighting network. This network then generates weights that adaptively select the most relevant features from the acoustic filterbank before they are passed to higher layers.
  2. Modulation Filterbank Learning: Following the acoustic stage, a second stage learns modulation filters. Similar to the acoustic stage, this also employs a relevance weighting sub-network. This sub-network receives embeddings of previous time steps and the output of the modulation filtering layer to generate weights that select relevant modulation features.

The core novelty lies in the feedback-based relevance weighting. Unlike traditional attention mechanisms, this approach applies weights as a form of soft feature selection without linear combination. The embeddings used for feedback are derived from a pre-trained "senone embedding network," which learns to represent phonetic states (senones) from the speech signal.

Datasets

The paper evaluates its approach on the following datasets:

  • Aurora-4: A challenging dataset for ASR under noisy conditions (additive noise with channel artifacts).
  • CHiME-3: A corpus featuring multi-microphone tablet device recordings in various everyday environments with both real and simulated noisy speech.
  • VOiCES: A large-scale speech dataset for ASR in complex environmental settings, characterized by reverberation and noise.
  • TIMIT: Used for visualizing the learned senone embeddings.

Key Insights and Ideas

  • Feedback is Crucial for Adaptability: Incorporating feedback from past predictions into the relevance weighting mechanism allows the model to dynamically adapt its feature selection based on the evolving acoustic context. This is a significant departure from static feature extraction.
  • Senone Embeddings Provide Meaningful Context: Pre-training an embedding network on senone targets (context-dependent triphone hidden Markov model states) provides rich phonetic information that can be effectively used as feedback for relevance weighting.
  • Two-Stage Relevance Weighting Captures Hierarchical Features: Applying relevance weighting at both the acoustic filterbank and modulation filterbank stages allows for a more comprehensive and adaptive learning of speech representations, capturing both low-level acoustic patterns and higher-level temporal dynamics.
  • Interpretability Through Feature Selection: The relevance weighting mechanism offers a degree of interpretability by explicitly highlighting which features are deemed important at different stages of processing.

Key Results

The proposed representation learning approach demonstrates significant improvements in ASR performance across multiple datasets:

  • Aurora-4: Achieved an average relative improvement of 15% over baseline mel filterbank features and 7% over baseline features on the CHiME-3 evaluation set. The best performing variant, using sigmoid activation and senone embeddings in both relevance weighting sub-networks, achieved a Word Error Rate (WER) of 9.1% on the Aurora-4 average.
  • CHiME-3: The 2-stage relevance weighting approach provided significant improvements, with an average relative improvement of 7% over MFB features in the evaluation set.
  • VOiCES: The model showed consistent improvements over baseline MFB features, indicating scalability to larger ASR tasks.
  • Cross-Domain Transfer: The learned acoustic filterbanks showed robustness and could be generalized to different datasets, suggesting the learned representations are not overly specific to the training domain.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, the multi-stage nature and feedback mechanisms likely increase computational complexity compared to simpler front-ends.
  • Future Work: The paper suggests that the learned relevance weighting is applicable to other learnable front-ends, hinting at potential extensions to existing methods. The robustness of the learned representations across domains also suggests further exploration of transfer learning for ASR.

Summary updated

BibTeX
@inproceedings{agrawal2021representation,
  author = {Purvi Agrawal and Sriram Ganapathy},
  title = {{Representation Learning for Speech Recognition Using Feedback Based Relevance Weighting}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9414649},
}
Figure