Out of Time: Automated Lip Sync in the Wild

Joon Son Chung, Andrew Zisserman

Abstract

The goal of this work is to determine the audio-video synchronisation between mouth motion and speech in a video. We propose a two-stream ConvNet architecture that enables the mapping between the sound and the mouth images to be trained end-to-end from unlabelled data. The trained network is used to determine the lip-sync error in a video. We apply the network to two further tasks: active speaker detection and lip reading. On both tasks we set a new state-of-the-art on standard benchmark datasets.
BibTeX
@inproceedings{chung2016out,
  author = {Joon Son Chung and Andrew Zisserman},
  title = {{Out of Time: Automated Lip Sync in the Wild}},
  booktitle = {ACCV Workshops},
  year = {2016},
  doi = {10.1007/978-3-319-54427-4\_19},
}

Loading related content...

Figure