Audio-guided self-supervised learning for disentangled visual speech representations
Lists
Only you see this
Abstract
In this paper, we propose a novel two-branch framework to learn the disentangled visual speech representations based on two particular observations. Its main idea is to introduce the audio signal to guide the learning of speech-relevant cues and introduce a bottleneck to restrict the speech-irrelevant branch from learning high-frequency and fine-grained speech cues. Experiments on both the word-level and sentence-level audio-visual speech datasets LRW and LRS2-BBC show the effectiveness. Our future work is to explore more explicit auxiliary tasks and constraints beyond the reconstruction task of the speech-relevant and irrelevant branch to improve further its ability of capturing speech cues in the video. Meanwhile, it’s also a nice try to combine multiple types of knowledge representations [10] to further boost the obtained speech epresentations, which is also left for the future work.
BibTeX
@article{feng2024audio,
author = {Dalu Feng and Shuang Yang and Shiguang Shan and Xilin Chen},
title = {{Audio-guided self-supervised learning for disentangled visual speech representations}},
journal = {Frontiers of Computer Science},
year = {2024},
url = {https://openreview.net/forum?id=zKl5L1qVKF},
}Loading related content...