MetaMixSpeech: Meta Task Augmentation for Low-Resource Speech Recognition

Yaqi Chen, Hao Zhang, Wenlin Zhang, XuKui Yang, Dan Qu, Yunpeng LiuEMNLP 2025

TL;DR

This paper introduces MetaMixSpeech, a novel meta-learning approach that tackles overfitting in low-resource speech recognition by augmenting both support and query sets with mixed features. This method significantly improves generalization across diverse languages and datasets, outperforming existing meta-learning techniques.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems have achieved remarkable success for widely spoken languages. However, a vast majority of the world's languages are considered "low-resource" due to a scarcity of transcribed speech data, pronunciation dictionaries, and language scripts. Building effective ASR for these languages is a significant challenge. Meta-learning has emerged as a promising paradigm to address this by enabling models to learn from multiple tasks (languages) and adapt quickly to new, unseen tasks. However, meta-learning itself faces challenges like "task overfitting" (where the model overfits to specific training tasks) and "learner overfitting" (where the meta-learner overfits to the meta-training data), hindering its generalization to new languages. This paper aims to mitigate these overfitting issues in multilingual meta-learning for ASR.

Methods

The core of MetaMixSpeech is a novel task augmentation strategy designed to combat overfitting. It operates in two phases:

  • Support Augmentation: To combat task overfitting, this method increases the diversity of training samples for a specific task. It achieves this by linearly combining perturbed features from the support set with the original features, and applying the same linear interpolation to their corresponding losses. This effectively creates "intermediate samples" that help the task learner generalize better.
  • Query Augmentation: To address learner overfitting and improve generalization to new tasks, this technique augments the query set. Similar to support augmentation, it mixes perturbed query set features with original features and their losses. This provides a broader meta-evaluation, enhancing the meta-learner's ability to generalize across different language families.

The paper leverages the MixSpeech method for this feature mixing, as direct mixing of speech data can be challenging due to variable lengths. The overall framework is model-agnostic and can be integrated with existing meta-learning algorithms.

Datasets

The experiments were conducted using two datasets:

  • Common Voice Corpus: Used for meta-training, with eight selected languages (Italian, German, Swedish, English, French, Dutch, Russian, and Portuguese), each providing approximately 5 hours of training data.
  • FLEURS Dataset: Used for evaluating generalization across diverse languages. Ten target languages were selected from FLEURS (Croatian, Maltese, Vietnamese, Finnish, Urdu, Nepali, Malayalam, Malay, Galician, and Kazakh).

Additionally, eight target languages from the Common Voice Corpus were used for evaluation (Breton, Irish, Romanian, Odia, Sorbian, Arabic, Ukrainian, and Czech).

Key Insights and Ideas

  • Overfitting is a critical bottleneck: The paper clearly identifies and analyzes two specific types of overfitting (task and learner) that plague multilingual meta-learning for ASR.
  • Augmentation is key to generalization: The proposed MetaMixSpeech demonstrates that strategically augmenting both support and query sets is crucial for improving a meta-learner's ability to generalize to unseen low-resource languages.
  • Query augmentation is particularly impactful: The results suggest that learner overfitting is a more significant issue than task overfitting in this context, and query augmentation plays a more substantial role in mitigating it. This is attributed to the diverse language families involved.
  • MetaMixSpeech offers a unified solution: By combining both support and query augmentation, MetaMixSpeech provides a comprehensive approach that effectively addresses both overfitting issues.

Key Results

  • MetaMixSpeech achieved a 6.35% improvement in Word Error Rate (WER) compared to standard meta-learning approaches on the FLEURS dataset.
  • The method demonstrated superior generalization capabilities across diverse datasets and language families, outperforming baselines like FT-Full, Vanilla-ASR, Multi-ASR, and Meta-ASR.
  • On the Common Voice dataset, MetaMixSpeech also showed robust performance, effectively mitigating overfitting issues observed in other methods.
  • Experiments with varying data proportions showed that MetaMixSpeech could achieve comparable performance to other methods with significantly less data, highlighting its efficiency in low-resource settings.

Limitations and Future Work

  • Limited meta-training data: The current work uses eight languages with five hours of data each for meta-training. The authors acknowledge that this might not fully capture the effects of long-tail data distributions.
  • Speech-centric focus: The experiments were limited to speech tasks. The authors suggest extending the method to other NLP tasks like machine translation and language generation.

Future work will involve expanding meta-training settings to account for long-tail distributions and applying MetaMixSpeech to a broader range of NLP tasks.

Summary updated

BibTeX
@inproceedings{chen2025metamixspeech,
  author = {Yaqi Chen and Hao Zhang and Wenlin Zhang and XuKui Yang and Dan Qu and Yunpeng Liu},
  title = {{MetaMixSpeech: Meta Task Augmentation for Low-Resource Speech Recognition}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure