Investigating Adapters for Parameter-efficient Low-resource Automatic Speech Recognition

Ahnaf Mozib Samin, Shekhar Nayak, Andrea De Marco, Claudia BorgNAACL 2025

TL;DR

This paper explores parameter-efficient "adapters" for automatic speech recognition (ASR) in low-resource languages, a crucial area given the vast number of endangered languages. They introduce a novel convolutional adapter (ConvAdapt) that outperforms standard adapters in low-data scenarios and show that adapters can achieve performance comparable to full model fine-tuning with significantly fewer trainable parameters.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) has seen tremendous progress, but this advancement is heavily skewed towards high-resource languages like English due to the availability of massive datasets. However, the world has over 7000 languages, many of which are low-resource or even zero-resource, facing the risk of extinction. While large pre-trained self-supervised models (like Wav2vec 2.0) offer a path to improve ASR for these languages through fine-tuning, this process is computationally expensive, resource-intensive, and can lead to "catastrophic forgetting" when dealing with multiple tasks or languages.

Parameter-efficient methods, particularly "adapters," have shown promise in natural language processing (NLP) by allowing models to adapt to new tasks without retraining the entire model. However, their application in ASR, especially for low-resource languages, remains less explored. This paper aims to bridge this gap by investigating how adapters can be effectively used for low-resource ASR, focusing on data requirements, transfer learning, and architectural choices.

Methods

The core idea is to insert small, trainable "adapter" modules into pre-trained ASR models, keeping the vast majority of the model's parameters frozen. This significantly reduces the number of parameters that need to be trained for a specific task or language.

The paper explores two main adapter architectures:

  • Bottleneck Adapters (FFAdapter): These are the standard adapters, consisting of two feed-forward layers that down-sample and then up-sample the model's internal representation. This is a well-established technique in NLP.
  • Convolutional Adapters (ConvAdapt): This is the novel contribution. Instead of feed-forward layers, ConvAdapt uses 1D convolutional layers. The authors hypothesize that the sparse connectivity and weight sharing inherent in convolutional networks can be more effective in low-data regimes compared to the fully connected nature of FF adapters.

They also investigate:

  • Data Requirements: How much data is needed for adapters to match full fine-tuning performance.
  • Transfer Learning: Whether pre-training adapters on a source language can benefit performance on a target language.
  • Adapter Architecture Scaling: The impact of increasing the number of feed-forward layers in bottleneck adapters.

Datasets

The experiments were conducted on three languages representing different language groups:

  • English: LibriSpeech corpus (standard 100-hour subset and LibriLight subsets).
  • Bengali: SUBAK.KO corpus.
  • Maltese: A combination of Common Voice, MASRI-HEADSET, MEP, Tube, MERLIN, and Parliament datasets.

Subsets of data were created ranging from 10 minutes to 50 hours to analyze data requirements. Standard development (dev) and test sets were used for evaluation.

Key Insights and Ideas

  • Adapters are Competitive, Not Always Superior in Few-Shot: While adapters can match full fine-tuning performance with sufficient data (around 10 hours), they are not as effective in extreme few-shot scenarios (10 minutes or 1 hour). This suggests that while adapters are parameter-efficient, they still require a minimum amount of data to learn meaningful representations.
  • ConvAdapt Shines in Low-Data Regimes: The proposed ConvAdapt significantly outperforms standard bottleneck adapters in extremely low-resource scenarios. This highlights the potential of convolutional structures for capturing relevant speech features with limited data.
  • Larger Models Benefit More from Adapters: Employing larger pre-trained models (like XLS-R 2B) with adapter tuning leads to better performance than fine-tuning smaller models, and even surpasses fine-tuning the large model itself in many cases. This suggests adapters act as a powerful regularizer, mitigating overfitting in large models.
  • Transfer Learning Potential: Pre-training adapters on a source language shows promise, especially for closely related languages, potentially enabling more efficient adaptation.

Key Results

  • Competitive Performance: With at least 10 hours of data, bottleneck adapters achieve performance comparable to full fine-tuning, while reducing trainable parameters by up to 31 times (e.g., from 2B to 64M for XLS-R 2B).
  • Few-Shot Advantage: ConvAdapt outperforms bottleneck adapters in extremely low-resource (10 min, 1 hour) scenarios, though still underperforming full fine-tuning.
  • Scaling Benefits: Increasing the number of feed-forward layers in bottleneck adapters improves performance, with 6 layers showing optimal results.
  • Language-Specific Performance: Adapter tuning shows the most benefits for English ASR. Bengali ASR performance with adapters is lower, potentially due to its limited representation in the pre-trained XLS-R model. However, with more Bengali data for adapter tuning, performance significantly improves.

Limitations and Future Work

  • Language Group Limitations: The transfer learning experiments used languages from distinct groups. The authors suggest that using closely related language pairs might yield greater performance gains.
  • Future Directions: The paper suggests exploring transfer learning in adapters with more closely related languages and investigating the application of adapters for multi-task learning using a single encoder.

Summary updated

BibTeX
@inproceedings{samin2025investigating,
  author = {Ahnaf Mozib Samin and Shekhar Nayak and Andrea De Marco and Claudia Borg},
  title = {{Investigating Adapters for Parameter-efficient Low-resource Automatic Speech Recognition}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure