Time-Domain Audio-Visual Speech Separation on Low Quality Videos

Yifei Wu, Chenda Li, Jinfeng Bai, Zhongqin Wu, Yanmin QianICASSP 2022 · OpenReview

Abstract

Incorporating visual information is a promising approach to improve the performance of speech separation. Many related works have been conducted and provide inspiring results. However, low quality videos appear commonly in real scenarios, which may significantly degrade the performance of normal audio-visual speech separation system. In this paper, we propose a new structure to fuse the audio and visual features, which uses the audio feature to select relevant visual features by utilizing the attention mechanism. A Conv-TasNet based model is combined with the proposed attention-based multi-modal fusion, trained with proper data augmentation and evaluated with 3 categories of low quality videos. The experimental results show that our system outperforms the baseline which simply concatenates the audio and visual features when training with normal or low quality data, and is robust to low quality video inputs at inference time.
BibTeX
@inproceedings{wu2022time,
  author = {Yifei Wu and Chenda Li and Jinfeng Bai and Zhongqin Wu and Yanmin Qian},
  title = {{Time-Domain Audio-Visual Speech Separation on Low Quality Videos}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=GorLlnGmGM},
}

Loading related content...

Figure