Language-Free Compositional Action Generation via Decoupling Refinement

Xiao Liu, Guangyi Chen, Yansong Tang, Guangrun Wang, Xiao-Ping Zhang, Ser-Nam Lim

This paper presents a novel approach to a difficult problem in computer vision: teaching machines to imagine complex, simultaneous human actions (like "walking while drinking") without ever seeing a combined example or reading a text description of one.

TL;DR

The authors propose a "language-free" framework that generates simultaneous 3D compositional actions by blending simple sub-actions based on their "motion energy." By utilizing a self-supervised refinement process involving 2D image inpainting, the model learns to synthesize realistic, unseen action combinations from basic building blocks without requiring expensive manual annotations.

What Problem Does This Paper Address?

Humans are naturally good at "compositional generation"—we can easily imagine or perform two actions at once, even if we've only ever practiced them separately. In the world of AI-driven 3D motion synthesis, however, this is incredibly difficult.

Most existing methods rely on language auxiliaries (like CLIP) to bridge the gap between different actions. While effective, this approach has two major drawbacks:

  1. High Cost: It requires massive amounts of labor-intensive text annotations.
  2. Sequential Bias: These models often default to temporal sequences (e.g., "walk, then run") rather than simultaneous actions (e.g., "walking while drinking").

The authors aim to break this dependence on text and generate simultaneous actions using only single-action data as a starting point.

Methods

The framework, illustrated in Figure 2, consists of three core stages:

  1. Action Coupling: To create training data for composite actions, the model blends two separate sub-actions. It uses an energy-based attention mask to identify which body parts are "active" for a specific action. For example, "walking" has high motion energy in the legs, while "drinking" has it in the arms. The model then couples these active parts to create a "pseudo-compositional" action.
  2. Conditional Action Generation (CVAE): The coupled motions are fed into a Conditional Variational Autoencoder. This model learns a latent space that can represent the variety and complexity of these blended motions, conditioned on the labels of the two sub-actions.
  3. Decoupling Refinement: Because the initial coupling can be "messy" or physically unrealistic, the authors introduce a refinement step. They project the 3D skeletons into 2D images and use a pre-trained Masked Autoencoder (MAE). By masking out the parts of the body associated with one sub-action, they task the MAE with "inpainting" or recovering the original sub-action. This forces the generative model to maintain semantic consistency—ensuring that the "walking" part of a "walking + drinking" motion still looks like walking.
The pipeline of the proposed framework, showing Action Coupling, CVAE generation, and Decoupling Refinement.

Datasets

Since no standard datasets existed for simultaneous compositional actions, the authors created two new benchmarks:

  • HumanAct-C: Derived from the HumanAct12 dataset, featuring 9 compositional categories (e.g., "walk+call", "sit+eat").
  • UESTC-C: Derived from the UESTC dataset, featuring 25 compositional categories based on 10 sub-actions.

In both cases, the training sets consist only of simple sub-actions, while the testing sets contain manually curated simultaneous compositions to evaluate the model's ability to generalize to unseen combinations.

Key Insights and Ideas

  • Body Part Independence: The core insight is that simultaneous actions usually involve disparate body parts. A person generally doesn't use their legs for two different distinct actions at the exact same time; they use legs for one (walking) and arms for another (drinking).
  • Motion Energy as a Semantic Proxy: Instead of needing a human to label which joints matter, the model simply looks at which joints move the most. This "motion energy" effectively identifies the "active" regions of an action.
  • Cross-Modal Refinement: Leveraging a 2D image model (MAE) to refine 3D motion is a clever way to use powerful, pre-trained self-supervised weights from the vision domain to solve a motion-domain problem.

Key Results

The model was evaluated using FID-C (a version of the Frechet Inception Distance tailored for compositional actions) and action recognition accuracy.

  • Quantitative Superiority: As shown in Table 1 (Figure 6), the proposed method significantly outperformed baselines like MotionCLIP and ACTOR+Mask. On the UESTC-C dataset, it achieved an FID of 90.13, compared to 121.27 for Latent Disentanglement and a staggering 2502.81 for MotionCLIP (which struggled without specific compositional training).
  • Qualitative Fluidity: Qualitative comparisons (Figure 4) show that while baseline methods often produce jittery or indistinct motions, the proposed method generates smooth, discernible actions where both sub-components are clearly visible.
Qualitative comparison showing the proposed method producing clearer simultaneous actions than baselines.

Limitations and Future Work

  • Physical Constraints: While the motions look semantically correct, the paper does not explicitly discuss physical constraints like balance or floor contact, which are common hurdles in 3D animation.
  • Complexity Limits: The current work focuses on coupling two actions. Scaling this to three or more simultaneous actions (e.g., "walking while drinking and nodding") remains an open challenge.
  • 2D Projection Loss: Projecting 3D data to 2D for the MAE refinement step inherently loses some spatial information, which might affect the precision of the refinement.

Overall, this research represents a significant step toward more efficient and autonomous 3D animation tools, moving away from the "data-hungry" requirements of traditional supervised learning.

Summary updated

Abstract

Composing simple actions into complex actions is crucial yet challenging. Existing methods largely rely on language annotations to discern composable latent semantics, which is costly and labor-intensive. In this study, we introduce a novel framework to generate compositional actions without language auxiliaries. Our approach consists of three components: Action Coupling, Conditional Action Generation, and Decoupling Refinement. Action Coupling integrates two sub-actions to generate pseudo-training examples. Then, a conditional generative model, CVAE is employed to facilitate the diverse generation. Decoupling Refinement leverages a self-supervised pre-trained model MAE to ensure semantic consistency between sub-actions and compositional actions. Due to the lack of existing datasets containing both sub-actions and compositional actions, we create two new datasets, named HumanAct-C and UESTC-C. Both qualitative and quantitative assessments are conducted to show our efficacy. 1
BibTeX
@inproceedings{liu2024language,
  author = {Xiao Liu and Guangyi Chen and Yansong Tang and Guangrun Wang and Xiao-Ping Zhang and Ser-Nam Lim},
  title = {{Language-Free Compositional Action Generation via Decoupling Refinement}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=MuPaH8RqoW},
}
Figure