Audio-guided self-supervised learning for disentangled visual speech representations

Dalu Feng, Shuang Yang, Shiguang Shan, Xilin ChenFrontiers of Computer Science 2024 · OpenReview

Abstract

In this paper, we propose a novel two-branch framework to learn the disentangled visual speech representations based on two particular observations. Its main idea is to introduce the audio signal to guide the learning of speech-relevant cues and introduce a bottleneck to restrict the speech-irrelevant branch from learning high-frequency and fine-grained speech cues. Experiments on both the word-level and sentence-level audio-visual speech datasets LRW and LRS2-BBC show the effectiveness. Our future work is to explore more explicit auxiliary tasks and constraints beyond the reconstruction task of the speech-relevant and irrelevant branch to improve further its ability of capturing speech cues in the video. Meanwhile, it’s also a nice try to combine multiple types of knowledge representations [10] to further boost the obtained speech epresentations, which is also left for the future work.
BibTeX
@article{feng2024audio,
  author = {Dalu Feng and Shuang Yang and Shiguang Shan and Xilin Chen},
  title = {{Audio-guided self-supervised learning for disentangled visual speech representations}},
  journal = {Frontiers of Computer Science},
  year = {2024},
  url = {https://openreview.net/forum?id=zKl5L1qVKF},
}

Loading related content...

Figure