Text-based 3D Motion Generation with a Mixture of Experts for Enhancing Performance

JaeYoung Seon, Seong-Eun Hong, KyeongMin Kim, SeungWoo Woo, Hyoseok Hwang, HyeongYeop KangJournal of the Korea Computer Graphics Society 2025 · OpenReview

Abstract

Traditional text-to-motion generation models often struggle with the inherent modularity of human movement, treating the body as a single holistic unit. This paper presents a novel architecture utilizing a Mixture of Experts (MoE) within a Transformer framework to capture fine-grained motion nuances. By employing sparse activation and a gating network, the model achieves a 30% reduction in training time and superior motion quality, as evidenced by an FID of 0.106 on the HumanML3D dataset.

Introduction

Generating realistic 3D human motion from natural language descriptions is a complex task requiring the coordination of multiple body parts. Standard "one-size-fits-all" models often fail to capture the independent rhythms of arms, legs, and torso, leading to "muddy" or uncoordinated animations. This research addresses these limitations by introducing a modular approach where specialized "expert" sub-networks handle different aspects of motion.

Methodology

The proposed system follows a two-stage generative process based on the T2M-GPT framework.

Overview of the proposed two-stage framework.

Motion Discretization

First, a Vector Quantized Variational Autoencoder (VQ-VAE) is used to discretize continuous 3D motion data into "motion tokens." This allows the generation process to be treated as a sequence prediction problem, similar to language modeling.

MoE-based Transformer

The core innovation is the replacement of standard feed-forward layers with MoE layers. Each layer contains multiple parallel experts. A Gating Network (router) determines which experts process each motion token using "top-k" routing (k=2). This sparse activation ensures that only a subset of the model is active at any time, maintaining computational efficiency while expanding total capacity.

Results and Discussion

The model was evaluated using the HumanML3D dataset, which contains 14,616 motion clips with corresponding skeletal data and text descriptions.

Quantitative Analysis

The MoE-based model achieved an FID of 0.106, outperforming the baseline T2M-GPT (0.116). Furthermore, the architecture enabled a 30% reduction in training time, converging in approximately 54 hours on an NVIDIA RTX A5000. Ablation studies identified 6 experts as the optimal configuration for balancing expressiveness and stability.

Qualitative Analysis

Visually, the MoE model demonstrates superior coordination in complex tasks. For instance, when generating motion for specific limb instructions, the model maintains a stable lower body while executing precise arm movements, avoiding the "sliding" artifacts common in holistic models.

Qualitative example of precise motion generation.

Conclusion

The application of Mixture of Experts to 3D motion generation significantly improves both performance and training efficiency. Future work will investigate explicit role assignment for experts and enhanced load-balancing techniques to further refine the coordination of generated human movements.

Summary updated

BibTeX
@article{seon2025text,
  author = {JaeYoung Seon and Seong-Eun Hong and KyeongMin Kim and SeungWoo Woo and Hyoseok Hwang and HyeongYeop Kang},
  title = {{Text-based 3D Motion Generation with a Mixture of Experts for Enhancing Performance}},
  journal = {Journal of the Korea Computer Graphics Society},
  year = {2025},
  url = {https://openreview.net/forum?id=BxGjsyKR3N},
}
Figure