The Pipeline System of ASR and NLU with MLM-based data Augmentation Toward Stop Low-Resource Challenge

Hayato Futami, Jessica Huynh, Siddhant Arora, Shih-Lun Wu, Yosuke Kashiwagi, Yifan Peng, Brian Yan, Emiru Tsunoo, Shinji Watanabe

Abstract

This paper describes our system for the low-resource domain adaptation track (Track 3) in Spoken Language Understanding Grand Challenge, which is a part of ICASSP Signal Processing Grand Challenge 2023. In the track, we adopt a pipeline approach of ASR and NLU. For ASR, we fine-tune Whisper for each domain with upsampling. For NLU, we fine-tune BART on all the Track3 data and then on low-resource domain data. We apply masked LM (MLM) -based data augmentation, where some of input tokens and corresponding target labels are replaced using MLM. We also apply a retrieval-based approach, where model input is augmented with similar training samples. As a result, we achieved exact match (EM) accuracy 63.3/75.0 (average: 69.15) for reminder/weather domain, and won the 1st place at the challenge.
BibTeX
@inproceedings{futami2023pipeline,
  author = {Hayato Futami and Jessica Huynh and Siddhant Arora and Shih-Lun Wu and Yosuke Kashiwagi and Yifan Peng and Brian Yan and Emiru Tsunoo and Shinji Watanabe},
  title = {{The Pipeline System of ASR and NLU with MLM-based data Augmentation Toward Stop Low-Resource Challenge}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10096049},
}
Figure