OhMG: Zero-shot Open-vocabulary Human Motion Generation

Junfan Lin, Jianlong Chang, Lingbo Liu, Guanbin Li, Liang Lin, Qi Tian, Chang Wen Chen

Introduction

Generating realistic 3D human motion from text descriptions is a significant challenge in computer vision, primarily due to the "data-starved" nature of the field. While massive datasets exist for text and images, paired datasets of 3D motion and descriptive text are relatively small and restricted to basic actions like "walking" or "running."

Existing approaches typically fall into two categories: supervised models that fail on open-vocabulary prompts (e.g., "fly like a bird") and online optimization methods like AvatarCLIP. While optimization-based methods handle diverse prompts, they are computationally expensive, often taking minutes to generate a single motion sequence, and can be unstable. OhMG (Offline Open-vocabulary Human Motion Generation) addresses these issues by providing an offline framework that generalizes to arbitrary text prompts instantly at inference time without requiring paired text-motion training data.

The OOHMG Framework

The core innovation of OhMG lies in its ability to perform zero-shot generation by bridging the gap between the CLIP latent space and 3D human poses. Instead of directly translating text to a full motion sequence, the system generates a single representative pose from the text, which then acts as a "prompt" for a pretrained motion generator.

Overview of the OOHMG framework showing the transition from text to a single pose prompt, which then drives the motion generator.

Architectural Pillars

  1. Motion Generator Pretraining: Inspired by masked language modeling, the authors pretrain a bidirectional transformer to reconstruct full motion sequences from randomly masked ones. During inference, this model is "prompted" with a single text-consistent pose, and it "fills in the blanks" to create fluid, physically plausible motion.
  2. Text-Pose Alignment (TPA): To connect text to 3D joints, the authors utilize CLIP (Contrastive Language-Image Pre-training). By rendering 3D poses into multi-view images, they distilled CLIP’s image-text knowledge into a dedicated 3D pose encoder. This allows the system to mathematically evaluate how well a 3D pose aligns with a specific text string.
  3. Wordless Training: This novel strategy allows the training of a Text-to-Pose generator without a massive text dataset. By sampling random vectors from the CLIP latent space and treating them as "pseudo-text" features, the generator learns to produce poses that align with the entire CLIP feature space. This ensures the model generalizes to real-world text effortlessly.

Results and Performance

OhMG achieves state-of-the-art performance in both alignment accuracy and generation speed. In Top-K R-precision tests, OhMG significantly outperformed existing baselines. For instance, its Top-1 accuracy reached 0.0792, a massive improvement over MotionCLIP (0.0029) and AvatarCLIP (0.0002).

Qualitatively, the motions generated by OhMG are more realistic and better aligned with complex semantic prompts. As shown in the comparison for the prompt "Eat a hamburger," OhMG produces a more recognizable and anatomically correct sequence than its competitors.

Qualitative comparison of motion generation for the prompt 'Eat a hamburger' across different models.

Furthermore, OhMG offers a massive efficiency advantage. Being an offline model, it is approximately 10,000x faster than optimization-based methods. It can process a batch of motions in roughly 0.0159 seconds, compared to the ~140 seconds required by AvatarCLIP for a single generation.

Conclusion and Future Work

OhMG demonstrates that "wordless training" and latent space alignment can overcome the limitations of small paired datasets in human motion generation. By treating motion generation as a prompting task, the framework achieves high-quality, open-vocabulary results with unprecedented speed. Future work may focus on "divide-and-conquer" strategies to handle complex, multi-stage temporal actions and incorporating more advanced foundation models to capture nuanced linguistic similes.

Summary updated

Abstract

Generating motion in line with text has attracted increasing attention nowadays. However, open-vocabulary human motion generation still remains touchless and undergoes the lack of diverse labeled data. The good news is that, recent studies of large foundation models (e.g., CLIP) have demonstrated superior performance on few/zero-shot image-text alignment, largely reducing the need for manually labeled data. In this paper, we take the advantage of CLIP for open-vocabulary 3D human motion generation in a zero-shot manner. Specifically, our model is composed of two stages, i.e., text2pose and pose2motion generations. For text2pose generation, to address the difficulty of optimization with direct supervision from CLIP, we propose to carve the versatile CLIP model into a slimmer but more specific model for aligning 3D poses and texts, via a novel pipeline distillation strategy. Optimizing with the distilled 3D pose-text model, we manage to concretize the text-pose knowledge of CLIP into a text2pose generator effectively and efficiently. As for pose2motion, drawing the inspiration of the advanced language model, we pretrain a transformer-based motion model, which makes up for the lack of motion dynamics of CLIP. After that, by formulating the condition poses as prompts, the motion generator can generate motions referring to the condition poses in a controllable and flexible manner.
BibTeX
@misc{lin2022ohmg,
  author = {Junfan Lin and Jianlong Chang and Lingbo Liu and Guanbin Li and Liang Lin and Qi Tian and Chang Wen Chen},
  title = {{OhMG: Zero-shot Open-vocabulary Human Motion Generation}},
  year = {2022},
  url = {https://openreview.net/forum?id=_NS69On3V_7},
}
Figure