Introduction
Generating realistic 3D human motion is a cornerstone of modern computer graphics and robotics. While significant progress has been made in generating single actions, teaching models to perform simultaneous compositional actions—such as "walking while drinking"—remains a significant challenge. Traditional approaches rely heavily on large-scale, hand-labeled datasets where motion is paired with precise text descriptions. Such data is both expensive and labor-intensive to produce.
Furthermore, existing models often leverage language-based latent semantics (e.g., via CLIP) to compose actions. While effective for sequential tasks ("walk, then run"), these methods frequently struggle with simultaneous compositions where multiple actions must be integrated into a single, fluid motion sequence. This paper introduces a framework for Language-free Compositional Action Generation via Decoupling Refinement, which learns to synthesize complex actions using only basic, single-action building blocks without requiring text annotations or examples of combined actions during training.
Methodology
The proposed framework addresses the compositionality problem through a tripartite architecture designed to learn elemental motion concepts and their interaction principles.
1. Action Coupling
To overcome the lack of compositional training data, the authors implement an "Action Coupling" stage. This process generates pseudo-compositional sequences by identifying the most active body parts for specific actions using an energy-based attention mask. For instance, legs are prioritized for "walking" while arms are prioritized for "drinking." These high-energy components are then blended to create a synthetic training target.
2. Conditional Action Generation
The core generative model is a Conditional Variational Autoencoder (CVAE). It learns a latent space where pairs of sub-action labels serve as conditioning variables. This allows the model to generate a unified motion sequence that attempts to satisfy both action requirements simultaneously.
3. Decoupling Refinement
A primary risk of mixing motions is the introduction of artifacts or semantic blurring. To mitigate this, the authors introduce a "Decoupling Refinement" stage. 3D skeletons are rendered into 2D images and processed by a pre-trained Masked Autoencoder (MAE). The MAE is tasked with recovering the original sub-actions from the composite motion. This "decoupling" requirement forces the generative model to maintain high motion quality and ensure that neither sub-action is lost in the composition.
Experiments and Results
The authors established two new benchmarks for this task: HumanAct-C (9 compositional categories) and UESTC-C (25 compositional categories). In both datasets, the training phase utilizes only single-action sequences, while the test phase evaluates performance on unseen simultaneous compositions.
Quantitative Performance
The proposed method demonstrates significant improvements over state-of-the-art baselines, including MotionCLIP and Latent Disentanglement.
As shown in the table above, the model achieves a Fréchet Inception Distance (FID) of 90.13 on UESTC-C, substantially lower than the 121.27 achieved by Latent Disentanglement. Notably, MotionCLIP failed to generalize effectively to these unseen compositions, resulting in an FID of 2502.81. The model also achieved high action recognition accuracy (87.20% on UESTC-C and 64.1% on HumanAct-C), confirming that the generated motions are semantically distinct and recognizable.
Conclusion
This research presents a robust method for generating simultaneous 3D actions without the need for expensive language supervision. By leveraging motion energy for coupling and self-supervised 2D models for decoupling refinement, the framework achieves high-quality, semantically consistent results. While the current approach excels when actions involve different body parts, future work may explore resolving conflicts when multiple actions compete for the same limbs, or extending the framework to more than two simultaneous actions.
Summary updated