Introduction
Generating realistic 3D human motion from text descriptions is a significant challenge in computer vision, primarily due to the "data-starved" nature of the field. While massive datasets exist for text and images, paired datasets of 3D motion and descriptive text are relatively small and restricted to basic actions like "walking" or "running."
Existing approaches typically fall into two categories: supervised models that fail on open-vocabulary prompts (e.g., "fly like a bird") and online optimization methods like AvatarCLIP. While optimization-based methods handle diverse prompts, they are computationally expensive, often taking minutes to generate a single motion sequence, and can be unstable. OhMG (Offline Open-vocabulary Human Motion Generation) addresses these issues by providing an offline framework that generalizes to arbitrary text prompts instantly at inference time without requiring paired text-motion training data.
The OOHMG Framework
The core innovation of OhMG lies in its ability to perform zero-shot generation by bridging the gap between the CLIP latent space and 3D human poses. Instead of directly translating text to a full motion sequence, the system generates a single representative pose from the text, which then acts as a "prompt" for a pretrained motion generator.
Architectural Pillars
- Motion Generator Pretraining: Inspired by masked language modeling, the authors pretrain a bidirectional transformer to reconstruct full motion sequences from randomly masked ones. During inference, this model is "prompted" with a single text-consistent pose, and it "fills in the blanks" to create fluid, physically plausible motion.
- Text-Pose Alignment (TPA): To connect text to 3D joints, the authors utilize CLIP (Contrastive Language-Image Pre-training). By rendering 3D poses into multi-view images, they distilled CLIP’s image-text knowledge into a dedicated 3D pose encoder. This allows the system to mathematically evaluate how well a 3D pose aligns with a specific text string.
- Wordless Training: This novel strategy allows the training of a Text-to-Pose generator without a massive text dataset. By sampling random vectors from the CLIP latent space and treating them as "pseudo-text" features, the generator learns to produce poses that align with the entire CLIP feature space. This ensures the model generalizes to real-world text effortlessly.
Results and Performance
OhMG achieves state-of-the-art performance in both alignment accuracy and generation speed. In Top-K R-precision tests, OhMG significantly outperformed existing baselines. For instance, its Top-1 accuracy reached 0.0792, a massive improvement over MotionCLIP (0.0029) and AvatarCLIP (0.0002).
Qualitatively, the motions generated by OhMG are more realistic and better aligned with complex semantic prompts. As shown in the comparison for the prompt "Eat a hamburger," OhMG produces a more recognizable and anatomically correct sequence than its competitors.
Furthermore, OhMG offers a massive efficiency advantage. Being an offline model, it is approximately 10,000x faster than optimization-based methods. It can process a batch of motions in roughly 0.0159 seconds, compared to the ~140 seconds required by AvatarCLIP for a single generation.
Conclusion and Future Work
OhMG demonstrates that "wordless training" and latent space alignment can overcome the limitations of small paired datasets in human motion generation. By treating motion generation as a prompting task, the framework achieves high-quality, open-vocabulary results with unprecedented speed. Future work may focus on "divide-and-conquer" strategies to handle complex, multi-stage temporal actions and incorporating more advanced foundation models to capture nuanced linguistic similes.
Summary updated