DATA2VEC-SG: Improving Self-Supervised Learning Representations for Speech Generation Tasks

Heming Wang, Yao Qian, Hemin Yang, Nauyuki Kanda, Peidong Wang, Takuya Yoshioka, Xiaofei Wang, Yiming Wang, Shujie Liu, Zhuo Chen, DeLiang Wang, Michael ZengICASSP 2023 · OpenReview

Abstract

Self-supervised learning has been successfully applied to various speech recognition and understanding tasks. However, for generative tasks such as speech enhancement and speech separation, most self-supervised speech representations did not show substantial improvements. To deal with this problem, in this paper, we propose data2vec-SG (Speech Generation), which is a teacher-student learning framework that addresses speech generation tasks. Our data2vec-SG introduces a reconstruction module into data2vec [1] and enforces the representations to contain not only the semantic information but also the acoustic knowledge to generate clean speech waveforms. Experimental results demonstrate that the proposed framework boosts the performance of various speech generation tasks including speech enhancement, speech separation, and packet loss concealment. Meanwhile, the learned representation is also capable of helping other downstream tasks, which is demonstrated by the good performance in the speech recognition task in both clean and noisy conditions.
BibTeX
@inproceedings{wang2023data,
  author = {Heming Wang and Yao Qian and Hemin Yang and Nauyuki Kanda and Peidong Wang and Takuya Yoshioka and Xiaofei Wang and Yiming Wang and Shujie Liu and Zhuo Chen and DeLiang Wang and Michael Zeng},
  title = {{DATA2VEC-SG: Improving Self-Supervised Learning Representations for Speech Generation Tasks}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=NrK0huy16Jz},
}

Loading related content...

Figure