OmniControl: Control Any Joint at Any Time for Human Motion Generation

Yiming Xie, Varun Jampani, Lei Zhong, Deqing Sun, Huaizu Jiang

Abstract

We present a novel approach named OmniControl for incorporating flexible spatial control signals into a text-conditioned human motion generation model based on the diffusion process. Unlike previous methods that can only control the pelvis trajectory, OmniControl can incorporate flexible spatial control signals over different joints at different times with only one model. Specifically, we propose analytic spatial guidance that ensures the generated motion can tightly conform to the input control signals. At the same time, realism guidance is introduced to refine all the joints to generate more coherent motion. Both the spatial and realism guidance are essential and they are highly complementary for balancing control accuracy and motion realism. By combining them, OmniControl generates motions that are realistic, coherent, and consistent with the spatial constraints. Experiments on HumanML3D and KIT-ML datasets show that OmniControl not only achieves significant improvement over state-of-the-art methods on pelvis control but also shows promising results when incorporating the constraints over other joints.

Introduction

Generating human motion from text descriptions, such as "a person picks up a cup," has advanced significantly with the advent of diffusion models. However, these models often act as "black boxes" regarding spatial precision. Achieving specific spatial goals—like ensuring a character's hand touches a precise coordinate at a specific frame—remains a challenge. Most existing models rely on relative pose representations (e.g., joint positions relative to the pelvis), which facilitates learning but complicates global "world" coordinate control. During generation, the pelvis position shifts, making it difficult to anchor other joints to fixed external points without causing physical artifacts like "stretchy" limbs or sliding feet.

Methodology: Dual-Guidance Framework

OmniControl addresses these challenges by building upon the Motion Diffusion Model (MDM) and introducing a dual-guidance system that balances mathematical accuracy with biological plausibility.

OmniControl allows for the simultaneous control of multiple joints, such as both wrists, and supports motion in-betweening.

Analytic Spatial Guidance

This module ensures that the generated motion strictly adheres to target coordinates. During the denoising process, the model's internal relative pose estimates are converted into global coordinates. The system calculates the error between these estimates and the desired spatial signals, using the resulting gradients to "nudge" the motion toward the target. A dynamic iterative strategy applies more intensive corrections in the final stages of denoising to ensure precision.

Realism Guidance

To prevent the "broken puppet" effect where moving one joint causes unnatural poses, Realism Guidance provides feature corrections. Inspired by ControlNet, this module is a trainable copy of the motion model's internal layers. It ensures that the entire body reacts coherently to spatial constraints, maintaining balance and natural movement across all joints.

Key Insights

  • Hybrid Representation: OmniControl maintains relative representations for fluid motion generation while performing on-the-fly global conversions for precise control.
  • Gradient-Based Optimization: Instead of hard-coding joint positions, which leads to jitter, the model uses error gradients to guide the diffusion process smoothly.
  • Universal Versatility: A single model can handle constraints on any combination of joints (head, feet, wrists, etc.) simultaneously, eliminating the need for task-specific models.

Results and Evaluation

The framework was evaluated on the HumanML3D and KIT-ML datasets, demonstrating a new state-of-the-art in controllable motion generation.

The model demonstrates robustness to sparse signals, successfully generating a complex "walk-sit-walk" sequence from a single middle-point constraint.
  • Accuracy: On HumanML3D, OmniControl reduced average joint error by 79.2% compared to the previous best method (GMD).
  • Realism: The model achieved a significantly lower FID score (0.218 vs. 0.475 for PriorMDM), indicating higher motion quality.
  • Sparsity Handling: The model can interpolate complex motions even when provided with minimal "keyframe" constraints, such as a single pelvis position in the middle of a sequence.

Conclusion and Future Work

OmniControl provides a flexible and precise method for human motion generation. While it excels in spatial accuracy, future work will address remaining issues such as "foot skating" by integrating physical constraints like gravity and friction. Additionally, optimizing inference speed—currently impacted by the iterative guidance process—and handling conflicting user constraints remain areas for further research.

Summary updated

Abstract

We present a novel approach named OmniControl for incorporating flexible spatial control signals into a text-conditioned human motion generation model based on the diffusion process. Unlike previous methods that can only control the pelvis trajectory, OmniControl can incorporate flexible spatial control signals over different joints at different times with only one model. Specifically, we propose analytic spatial guidance that ensures the generated motion can tightly conform to the input control signals. At the same time, realism guidance is introduced to refine all the joints to generate more coherent motion. Both the spatial and realism guidance are essential and they are highly complementary for balancing control accuracy and motion realism. By combining them, OmniControl generates motions that are realistic, coherent, and consistent with the spatial constraints. Experiments on HumanML3D and KIT-ML datasets show that OmniControl not only achieves significant improvement over state-of-the-art methods on pelvis control but also shows promising results when incorporating the constraints over other joints.
BibTeX
@inproceedings{xie2024omnicontrol,
  author = {Yiming Xie and Varun Jampani and Lei Zhong and Deqing Sun and Huaizu Jiang},
  title = {{OmniControl: Control Any Joint at Any Time for Human Motion Generation}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=gd0lAEtWso},
}
Figure