Cross-Conditioned Recurrent Networks for Long-Term Synthesis of Inter-Person Human Motion Interactions

Jogendra Nath Kundu, Himanshu Buckchash, Priyanka Mandikal, Rahul M. V., Anirudh Jamkhandi, Venkatesh Babu R.WACV 2020 · OpenReview

Abstract

Modeling dynamics of human motion is one of the most challenging sequence modeling problems, with diverse applications in the animation industry, human-robot interaction, motion-based surveillance, etc. Available attempts to use auto-regressive techniques for long-term single-person motion generation usually fail, resulting in stagnated motion or divergence to unrealistic pose patterns. In this paper, we propose a novel cross-conditioned recurrent framework targeting long-term synthesis of inter-person interactions beyond several minutes. We carefully integrate positive implications of both auto-regressive and encoder-decoder recurrent architectures, by interchangeably utilizing two separate fixed-length cross-person motion prediction models for long-term generation in a novel hierarchical fashion. As opposed to prior approaches, we guarantee structural plausibility of 3D pose by training the recurrent model to regress latent representation of a separately trained generative pose embedding network. Different variants of the proposed frameworks are evaluated through extensive experiments on SBU-interaction, CMU-MoCAP, and an in-house collection of duet-dance datasets. Qualitative and quantitative evaluation on several tasks, such as short-term motion prediction, long-term motion synthesis, and interaction-based motion retrieval against prior state-of-the-art approaches clearly highlight the superiority of the proposed framework.

Introduction

Modeling human motion is a fundamental challenge in computer vision, with applications ranging from realistic animation to human-robot interaction. While short-term prediction (e.g., the next 500ms of a walk) has seen significant progress, long-term synthesis—generating realistic motion over several minutes—remains elusive. Traditional auto-regressive models often suffer from two primary issues: motion stagnancy, where characters freeze into a static mean pose, or divergence, where skeletons deform into unrealistic shapes. Furthermore, most research focuses on single-person motion, neglecting the complex, synchronized dynamics of two-person interactions like duet dancing.

Methodology

The proposed framework, Cross-Conditioned Recurrent Networks (ccRNN), addresses these challenges through a multi-stage architectural approach designed for stability and structural integrity.

Generative Pose Embedding

To ensure that generated motions remain kinematically plausible, the authors avoid predicting raw 3D joint coordinates directly. Instead, they utilize an Adversarial Auto-Encoder (AAE) to learn a view-invariant latent representation ($z$) of human poses.

The framework for learning pose embeddings and coordinate transformations to ensure structural plausibility.

By training the recurrent model to regress these latent codes, the system guarantees that the decoded output always corresponds to a valid human pose, effectively preventing the "noodle-like" artifacts common in raw coordinate regression.

Cross-Conditioned RNN (ccRNN)

The core of the interaction modeling is the ccRNN, which mimics a "leader-follower" dynamic. It consists of two sub-networks:

  • pRNN (Prediction): Maps Person A’s motion to the corresponding motion for Person B.
  • fRNN (Forecasting): Predicts Person B’s future states based on the context provided by the pRNN.

Hierarchical Synthesis (ccRNN-syn)

For long-term generation, the authors introduce a hierarchical auto-regression scheme. By interchangeably utilizing models trained for both directions (Person A $\rightarrow$ B and Person B $\rightarrow$ A), the framework regularizes the generation process. This prevents the error accumulation that typically leads to motion freezing in long sequences, enabling synthesis for several minutes.

Datasets and Evaluation

The model was evaluated on the SBU-Interaction and CMU-MoCAP datasets, as well as a new DuetDance dataset. This in-house collection contains complex, long-term duet performances (e.g., Salsa, Samba) processed into 3D skeletons from YouTube videos. Evaluation included short-term accuracy metrics and a "Critic Model" (a discriminator) to assess the realism of long-term synthesis. If a trained classifier cannot distinguish between real and synthesized motion, the synthesis is considered successful.

Results

The ccRNN framework demonstrates superior performance in both short-term prediction and long-term stability. While baseline models like acLSTM tend to freeze into static poses after approximately 10-20 seconds, the proposed ccRNN-syn generates coordinated, lively interactions for over 300 seconds.

Qualitative comparison showing that while baseline models like acLSTM stagnate over time (top), the proposed ccRNN variants maintain dynamic, realistic motion for over 60 seconds.

Quantitatively, the "Critic" test showed that synthesized motion from the proposed model was significantly harder to distinguish from real data (accuracy of 0.64 at 20 seconds) compared to standard models (accuracy of 0.86).

Conclusion and Future Work

This work presents a robust framework for long-term inter-person motion synthesis. By combining latent pose embeddings with cross-conditioned recurrent architectures, the model achieves unprecedented stability in generating complex interactions. Future directions include incorporating stochastic modeling to capture the inherent variability of human movement and utilizing RNN-based adversarial losses to further enhance motion "liveliness."

Summary updated

Abstract

Modeling dynamics of human motion is one of the most challenging sequence modeling problem, with diverse applications in animation industry, human-robot interaction, motion-based surveillance, etc. Available attempts to use auto-regressive techniques for long-term single-person motion generation usually fails, resulting in stagnated motion or divergence to unrealistic pose patterns. In this paper, we propose a novel cross-conditioned recurrent framework targeting long-term synthesis of inter-person interactions beyond several minutes. We carefully integrate positive implications of both auto-regressive and encoder-decoder recurrent architecture, by interchangeably utilizing two separate fixed-length cross person motion prediction models for long-term generation in a novel hierarchical fashion. As opposed to prior approaches, we guarantee structural plausibility of 3D pose by training the recurrent model to regress latent representation of a separately trained generative pose embedding network. Different variants of the proposed frameworks are evaluated through extensive experiments on SBU-interaction, CMU-MoCAP and an inhouse collection of duet-dance dataset. Qualitative and quantitative evaluation on several tasks, such as Short-term motion prediction, Long-term motion synthesis and Interaction-based motion retrieval against prior state-of-the-art approaches clearly highlight superiority of the proposed framework.
BibTeX
@inproceedings{kundu2020cross,
  author = {Jogendra Nath Kundu and Himanshu Buckchash and Priyanka Mandikal and Rahul M. V. and Anirudh Jamkhandi and Venkatesh Babu R.},
  title = {{Cross-Conditioned Recurrent Networks for Long-Term Synthesis of Inter-Person Human Motion Interactions}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=IZZ8d4w9nEd},
}
Figure