Speech Data Selection for Efficient ASR Fine-Tuning using Domain Classifier and Pseudo-Label Filtering

Pradeep Rangappa, Juan Zuluaga-Gomez, Srikanth R. Madikeri, Andrés Carofilis, Jeena J. Prakash, Sergio Burdisso, Shashi Kumar, Esaú Villatoro-Tello, Iuliia Nigmatulina, Petr Motlícek, Karthik Pandia, Aravind GanapathirajuICASSP 2025 · OpenReview

Abstract

In real-world speech data processing, the scarcity of annotated data and the abundance of unlabelled speech data present a significant challenge. To address this, we propose an efficient data selection pipeline for fine-tuning ASR models by generating pseudo-labels using WhisperX pipeline and selecting efficient labels for fine-tuning. In our work, we propose a domain classifier system developed with a computationally inexpensive TFIDF and classical machine learning algorithm. Later, we filter data from the classifier output using a novel metric that assesses word ratio and perplexity distribution. The filtered pseudo labels are then used for fine-tuning standard encoder-decoder Whisper models and Zipformer. Our proposed data selection pipeline reduces the dataset size by approximately 1/100th while maintaining performance comparable to the full dataset, outperforming random domain-independent selection strategies.
BibTeX
@inproceedings{rangappa2025speech,
  author = {Pradeep Rangappa and Juan Zuluaga-Gomez and Srikanth R. Madikeri and Andrés Carofilis and Jeena J. Prakash and Sergio Burdisso and Shashi Kumar and Esaú Villatoro-Tello and Iuliia Nigmatulina and Petr Motlícek and Karthik Pandia and Aravind Ganapathiraju},
  title = {{Speech Data Selection for Efficient ASR Fine-Tuning using Domain Classifier and Pseudo-Label Filtering}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=89uFwiZy8r},
}

Loading related content...

Figure