Free-View Expressive Talking Head Video Editing

Yuantian Huang, Satoshi Iizuka, Kazuhiro FukuiICASSP 2023 · OpenReview

Abstract

We present a novel framework for talking head video editing, allowing users to freely edit head pose, emotion, and eye blink while maintaining audio-visual synchronization. Unlike previous approaches that mainly focus on generating a talking head video, our proposed model is able to edit the talking heads of an input video and restore it to full frames, which supports a broader range of applications. Our proposed framework consists of two parts: a) a reconstruction-based generator that can generate talking heads fitting to the original frame while corresponding to freely controllable attributes, including head pose, emotion, and eye blink. b) a multiple-attribute discriminator that enforces attribute-visual synchronization. We additionally introduce attention modules and perceptual loss to improve the overall generation quality. We compare existing approaches as corroborated by quantitative metrics and qualitative comparisons.
BibTeX
@inproceedings{huang2023free,
  author = {Yuantian Huang and Satoshi Iizuka and Kazuhiro Fukui},
  title = {{Free-View Expressive Talking Head Video Editing}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=m2mgc3OxVS},
}

Loading related content...

Figure