Continual Learning for Personalized Co-Speech Gesture Generation

Chaitanya Ahuja, Pratik Joshi, Ryo Ishii, Louis-Philippe MorencyICCV 2023 · DOI

Abstract

Embodied AI agents require the ability to mimic personalized human gestures to appear natural. However, learning these styles sequentially often leads to catastrophic forgetting of previously mastered styles, especially in low-resource settings. We present C-DiffGAN, a framework designed for continual, crossmodal learning of co-speech gestures. By utilizing generative reminiscences of previous speakers and a novel crossmodal alignment loss, C-DiffGAN retains old gesture styles while adapting to new speakers with as little as two minutes of data. Our results show a significant reduction in forgetting (over 15x) and improved generation accuracy compared to existing state-of-the-art methods.

Introduction

Human communication is inherently multimodal, where spoken language is accompanied by personalized rhythmic gestures. For digital assistants and robots to interact naturally, they must adapt to the unique "body language" of individual users. Traditional models require simultaneous training on massive datasets, which is impractical for real-world scenarios where agents encounter new speakers sequentially.

Fine-tuning a pre-trained model on a new individual typically results in catastrophic forgetting, where the model loses the complex relationship between speech audio and skeletal motion for previous speakers. This challenge is exacerbated in "low-resource" environments where only a few minutes of data are available for a new target speaker.

Methods

The C-DiffGAN framework addresses these challenges through three integrated components:

  1. Low-resource Reminiscences: To avoid storing large volumes of sensitive video data, the model uses its previous state to generate synthetic gestures ("reminiscences") for earlier speakers. These act as a memory buffer for rehearsal without the need for raw data storage.
  2. Crossmodal Alignment ($L_{ccf}$): During the adaptation to a new speaker, this loss function enforces consistency between current outputs and the generated reminiscences. This ensures that the mapping from speech to motion for prior speakers remains intact.
  3. Efficient Adaptation ($L_{diffgan}$): To prevent overfitting on limited data (e.g., 2–10 minutes), the model updates only a single layer of the source model. This targeted parameter update captures the specific crossmodal shifts of the new speaker while maintaining the general human motion patterns learned previously.
The C-DiffGAN framework architecture showing the generation of reminiscences and crossmodal alignment.

Experimental Setup

The model was evaluated using the PATS dataset, comprising 25 speakers with diverse gesturing styles. To simulate real-world constraints, training for target speakers was limited to 2 or 10 minutes. C-DiffGAN was compared against several baselines, including DiffGAN (standard adaptation) and MeRGAN (continual learning).

Key Results

C-DiffGAN demonstrates superior performance in both accuracy and retention:

  • Generation Quality: C-DiffGAN achieved a Fréchet Inception Distance (FID) of 56.2, significantly lower than the 613.6 observed in standard DiffGAN adaptation, indicating much higher realism.
  • Retention of Prior Knowledge: The model reduced average forgetting by more than 15x compared to MeRGAN baselines. As shown in the forgetting curves, C-DiffGAN maintains high performance across successive "experiences" (new speakers), whereas other models show sharp declines.
  • User Evaluation: Participants in a user study rated C-DiffGAN gestures as more natural and better at preserving individual speaker identities, nearly matching "Buffer Replay" methods that require storing actual historical data.
Comparison of forgetting rates across different models as they encounter new speakers (experiences).

Conclusion and Future Work

C-DiffGAN provides a robust solution for the sequential learning of personalized gestures in low-resource settings. By leveraging generative reminiscences, it effectively mitigates crossmodal catastrophic forgetting. Future research will focus on scaling this approach to hundreds of speakers, incorporating finer motion details like facial expressions, and integrating environmental context to further enhance the realism of embodied agents.

Summary updated

BibTeX
@inproceedings{ahuja2023continual,
  author = {Chaitanya Ahuja and Pratik Joshi and Ryo Ishii and Louis-Philippe Morency},
  title = {{Continual Learning for Personalized Co-Speech Gesture Generation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01910},
}
Figure