Implicit Compositional Generative Network for Length-Variable Co-Speech Gesture Synthesis

Chenghao Xu, Jiexi Yan, Yanhua Yang, Cheng Deng

Abstract

Co-speech gesture synthesis is a practical yet challenging task that aims to generate body motion sequences in line with speech audio. Most of the existing methods can only generate the gesture sequence with a fixed number of frames, which does not satisfy the high-quality requirement of the virtual speech video in real-world applications. In this paper, we propose a novel Implicit Compositional Generative Network (ICGN) for length-variable co-speech gesture synthesis. In ICGN, the implicit neural representation is captured and optimized for a whole gesture sequence of arbitrary length with temporal embeddings. Moreover, to enforce the synthesized gestures more realistic and consistent, we compositionally generate the gesture sequence through a well-designed asymmetric two-stream network that effectively captures and utilizes the rich correlations between speech audio and human body motions. In this way, the coarse and fine-grained gestures are synthesized, respectively, according to the corresponding content-aware and emotion-aware audio components. Extensive experiments on four widely-used benchmarks demonstrate that the proposed method renders realistic human gestures and achieves the superior performance against several state-of-the-art methods.
BibTeX
@article{xu2024implicit,
  author = {Chenghao Xu and Jiexi Yan and Yanhua Yang and Cheng Deng},
  title = {{Implicit Compositional Generative Network for Length-Variable Co-Speech Gesture Synthesis}},
  journal = {IEEE Trans. Multimedia},
  year = {2024},
  url = {https://openreview.net/forum?id=YXVzeCbpHA},
}

Loading related content...

Figure