Data Techniques For Online End-to-end Speech Recognition

Yang Chen, Weiran Wang, I-Fan Chen, Chao WangarXiv 2020 · OpenReview

Abstract

Practitioners often need to build ASR systems for new use cases in a short amount of time, given limited in-domain data. While recently developed end-to-end methods largely simplify the modeling pipelines, they still suffer from the data sparsity issue. In this work, we explore a few simple-to-implement techniques for building online ASR systems in an end-to-end fashion, with a small amount of transcribed data in the target domain. These techniques include data augmentation in the target domain, domain adaptation using models previously trained on a large source domain, and knowledge distillation on non-transcribed target domain data, using an adapted bi-directional model as the teacher; they are applicable in real scenarios with different types of resources. Our experiments demonstrate that each technique is independently useful in the improvement of the online ASR performance in the target domain.

Introduction

End-to-end (E2E) speech recognition has significantly simplified the modeling pipeline by eliminating the need for complex, hand-engineered alignments. However, these models are notoriously data-hungry, creating a "cold start" problem for new domains where only a few hours of transcribed audio may be available. Furthermore, high-performing E2E models often rely on bi-directional architectures that require the entire utterance to be available, making them unsuitable for low-latency, online ASR. This paper investigates how to bridge the performance gap between offline bi-directional models and online uni-directional models using three simple-to-implement data-centric techniques.

Methodology

The proposed framework relies on three primary pillars:

1. Data Augmentation

To increase the diversity of the small target dataset, the authors employ two main techniques:

  • Speed Perturbation: Artificially altering the speed of the audio (e.g., factors of 0.9 and 1.1). Uniquely, this is implemented by resizing the spectrogram like an image, which is more computationally efficient than re-processing raw audio.
  • Spectral Masking: Randomly masking blocks of frequency or time in the spectrogram to force the model to learn more robust features.
Visual representation of data augmentation techniques, including speed perturbation and spectral masking.

2. Domain Adaptation

The system leverages a model pre-trained on a massive source domain (15,000 hours of Amazon Transcribe data). This model is then fine-tuned on the target domain. To facilitate rapid adjustment to new acoustic environments, Linear Input Networks (LIN) are utilized—adding a simple adapter layer between the input features and the pre-trained model.

3. Knowledge Distillation (Teacher-Student)

This technique uses a powerful bi-directional "Teacher" model trained on the target data. Since the Teacher is not restricted by real-time constraints, it achieves superior performance. This Teacher generates pseudo-labels for a large set of unlabeled audio. A uni-directional "Student" model is then trained to mimic the Teacher’s output on this unlabeled data, effectively transferring the benefits of future context without requiring it at runtime.

Experimental Results

The effectiveness of these techniques was evaluated on the Wall Street Journal (WSJ) benchmark and a challenging in-house conversational dataset.

Impact of Augmentation

Initial experiments on the WSJ dataset showed that both speed perturbation and spectral masking provide independent gains. As shown in the table below, combining both augmentation methods yields the best performance on development and evaluation sets.

Performance comparison (WER) of a model trained on target data with and without augmentation techniques.

Combined Performance

The cumulative effect of these techniques is substantial:

  • WSJ Dataset: The baseline uni-directional model's Word Error Rate (WER) of 17.72% was reduced to 7.58% after applying adaptation, augmentation, and distillation—a relative improvement of over 57%.
  • Conversational Speech: Starting from a pre-trained model (27.68% WER), the proposed techniques brought the WER down to 20.64%, representing a 25.4% relative improvement.

Conclusion

This work demonstrates that simple-to-implement data techniques—augmentation, domain adaptation, and teacher-student distillation—can significantly improve the performance of online E2E ASR systems in low-resource scenarios. The gains from these methods are additive, showing that a data-centric approach can effectively compensate for architectural limitations in real-time applications.

Summary updated

Abstract

Practitioners often need to build ASR systems for new use cases in a short amount of time, given limited in-domain data. While recently developed end-to-end methods largely simplify the modeling pipelines, they still suffer from the data sparsity issue. In this work, we explore a few simple-to-implement techniques for building online ASR systems in an end-to-end fashion, with a small amount of transcribed data in the target domain. These techniques include data augmentation in the target domain, domain adaptation using models previously trained on a large source domain, and knowledge distillation on non-transcribed target domain data, using an adapted bi-directional model as the teacher; they are applicable in real scenarios with different types of resources. Our experiments demonstrate that each technique is independently useful in the improvement of the online ASR performance in the target domain.
BibTeX
@misc{chen2020data,
  author = {Yang Chen and Weiran Wang and I-Fan Chen and Chao Wang},
  title = {{Data Techniques For Online End-to-end Speech Recognition}},
  year = {2020},
  eprint = {2001.09221},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2001.09221},
}
Figure