Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented Speech

Katrin Tomanek, Vicky Zayats, Dirk Padfield, Kara Vaillancourt, Fadi BiadsyEMNLP 2021 · DOI

Introduction

Standard Automatic Speech Recognition (ASR) systems are typically trained on vast amounts of "canonical" speech, characterized by standard accents and clear articulation. While these systems perform exceptionally well for the majority of users, they often fail those at the "tail" of the distribution—individuals with heavy accents or atypical speech patterns resulting from conditions such as ALS, stroke, or cerebral palsy.

Personalizing ASR models through full-model fine-tuning can significantly improve performance for these users. However, this approach is difficult to scale. Modern ASR models can reach sizes of several gigabytes; storing and deploying a unique, full-sized model for every individual user or accent group is logistically and financially prohibitive for both cloud-based and on-device applications. This paper explores residual adapters as a highly parameter-efficient alternative for ASR adaptation.

Methods: Residual Adapter Architecture

The researchers propose the use of residual adapters—small bottleneck layers injected into a frozen, pre-trained speech recognition model. The architecture of these adapters is designed for maximum efficiency:

  1. Bottleneck Structure: The input is first projected down to a significantly smaller dimension ($d_b$), passed through a non-linear ReLU activation, and then projected back up to the original dimension.
  2. Integration: These modules are integrated into the encoder layers of state-of-the-art architectures, specifically the Recurrent Neural Transducer (RNN-T) and the Transformer Transducer (T-T).
  3. Frozen Backbone: During the adaptation process, the entire pre-trained model remains frozen. Only the parameters within the tiny residual adapter layers are updated.
Overview of the RNN-T/T-T architecture and the integration of residual adapters into the Transformer encoder.

By using a bottleneck dimension as small as 16, the authors achieved a 100x reduction in updated parameters compared to fine-tuning the full encoder. In the RNN-T model, this meant updating only 197K parameters instead of 98.7M.

Experimental Setup

The study evaluated the effectiveness of residual adapters using two primary datasets:

  • Euphonia Corpus: A large-scale dataset of atypical speech from over 1,000 speakers with various impairments. A diverse subset of 100 speakers was used for this study.
  • Mozilla Common Voice: Used to test adaptation to 10 different English accents, including those from India, Australia, Scotland, and the Philippines.

Key Results and Insights

Performance Parity with Full Fine-Tuning

The most significant finding is that updating less than 0.5% of the model's parameters yields results nearly identical to full-model fine-tuning. For atypical speech, full encoder fine-tuning resulted in an 80% relative Word Error Rate (WER) improvement, while residual adapters achieved a 77% relative improvement. This demonstrates that the vast majority of the model's knowledge remains relevant, and only minor adjustments are needed to handle specialized speech patterns.

Robustness Across Impairment Severity

Residual adapters proved effective not just for mild speech impairments but across the entire spectrum of severity. The relative improvements remained consistent for mild, moderate, and severe cases, suggesting that the bottleneck layers are capable of capturing complex phonetic distortions.

Distribution of adapter performance metrics across mild, moderate, and severe speech impairment levels.

Efficiency and Latency

Beyond parameter efficiency, the use of adapters led to a 40% speedup in training time due to the drastically reduced number of gradient calculations. Crucially, the addition of these small layers did not result in any measurable increase in inference latency, making them suitable for real-time, on-device applications.

Conclusion

Residual adapters provide a scalable and efficient solution for personalizing ASR systems. By enabling high-quality speech recognition for atypical and accented speakers with minimal storage and computational overhead, this technology paves the way for more inclusive and accessible AI. Future work will investigate optimal adapter placement within the encoder and explore dynamic adaptation strategies for progressive neurological conditions.

Summary updated

BibTeX
@inproceedings{tomanek2021residual,
  author = {Katrin Tomanek and Vicky Zayats and Dirk Padfield and Kara Vaillancourt and Fadi Biadsy},
  title = {{Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented Speech}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.541},
}
Figure