ReMoGPT: Part-Level Retrieval-Augmented Motion-Language Models

Qing Yu, Mikihiro Tanaka, Kent Fujiwara

TL;DR

This paper introduces ReMoGPT, a novel motion-language model that tackles the challenge of generating diverse and rare human motions. By incorporating a fine-grained, body-part-level retrieval mechanism, ReMoGPT can access and leverage relevant motion examples from a database, significantly improving generation quality and generalizability.

What Problem Does This Paper Address?

Generating realistic and diverse 3D human motion from text descriptions is crucial for applications like gaming, robotics, and virtual reality. While large language models (LLMs) have made strides in various domains, creating models that can handle the nuances of human motion, especially less common or "rare" movements, remains a significant challenge. Existing methods often struggle with diversity and generalizability due to limited training data and the inherent complexity of motion. They tend to perform well on common motions but falter when encountering less frequent ones. This paper aims to bridge this gap by developing a unified model that can perform multiple motion-related tasks (generation, captioning, retrieval) and excel even with rare motion data.

Methods

ReMoGPT is built upon a transformer-based motion-language model, similar to MotionGPT, but with a key enhancement: retrieval augmentation. The core innovation lies in their Part-Level Text-Motion Retrieval (PL-TMR). Instead of treating the entire body as a single unit for retrieval, PL-TMR breaks down motion into distinct body parts (e.g., arms, legs, backbone). Each part is processed by a dedicated lightweight transformer encoder, creating fine-grained embeddings. These part-level embeddings are then used to compute similarity scores between text descriptions and specific motion segments. This approach allows for more precise retrieval of relevant motion examples, especially when the text description focuses on particular body actions.

The retrieved motion-text pairs are then incorporated into the prompt for the main motion-language model. This "retrieval-augmented generation" guides the model, helping it to generate more accurate and diverse motions. The model is trained using prompt-based question-and-answer tasks and instruction tuning, learning from the retrieved context to improve its performance across various motion-related tasks.

Datasets

The paper evaluates ReMoGPT on two standard text-to-motion datasets:

  • HumanML3D: Contains 14,616 motion clips from the AMASS dataset with 44,970 textual descriptions.
  • Motion-X: A larger, more recent dataset with over 81,051 motion clips and captions.

The training set of each dataset is used as the external database for the retrieval augmentation.

Key Insights and Ideas

  • Body-part level matters: The paper demonstrates that fine-grained retrieval based on body parts significantly outperforms whole-body retrieval for motion tasks. This is because specific actions are often localized to certain body parts, and this level of detail is crucial for accurate matching.
  • Retrieval as a knowledge boost: Augmenting LLMs with retrieved examples from a database is a powerful technique for improving performance, especially for tasks with limited data or a need for diversity. ReMoGPT shows this principle extends effectively to the domain of human motion.
  • Unified model for multiple tasks: ReMoGPT's ability to handle text-to-motion generation, motion-to-text captioning, and motion retrieval within a single framework highlights the potential of unified motion-language models.
  • Handling the "rare" motions: The model's success in generating rare motions, which are often underrepresented in datasets, is a significant achievement, suggesting better generalizability and robustness.

Key Results

  • State-of-the-art performance: ReMoGPT achieves state-of-the-art results on several benchmarks, including HumanML3D and Motion-X, for both text-to-motion generation and motion-to-text captioning.
  • Improved retrieval: The PL-TMR method consistently outperforms existing text-to-motion and motion-to-text retrieval methods, showing higher recall and lower median ranks.
  • Enhanced generation quality: ReMoGPT significantly improves motion generation quality, particularly for rare motion samples, demonstrating a substantial leap in diversity and generalizability compared to prior methods like MotionGPT and ReMoDiffuse. For instance, on HumanML3D, ReMoGPT achieves better FID, MMDist, and Diversity scores.
  • Qualitative improvements: Visualizations show ReMoGPT generating more accurate and contextually relevant motions, especially for novel or complex text prompts, where other models struggle.

Limitations and Future Work

The authors acknowledge that ReMoGPT still faces limitations in generating completely novel motions that are significantly different from the training data. This is attributed to the limited generalization ability of the motion tokenizer, which may struggle to reconstruct unseen motion patterns. They suggest that increasing the volume of training data could be a potential solution. Future work could also explore more advanced retrieval strategies or more robust motion tokenization techniques to further enhance the model's ability to handle entirely new motion categories.

Summary updated

Abstract

Generation of 3D human motion holds significant importance in the creative industry. While recent notable advances have been made in generating common motions, existing methods struggle to generate diverse and rare motions due to the complexity of motions and limited training data. This work introduces ReMoGPT, a unified motion-language generative model that solves a wide range of motion-related tasks by incorporating a multi-modal retrieval mechanism into the generation process to address the limitations of existing models, namely diversity and generalizability. We propose to focus on body-part-level motion features to enable fine-grained text-motion retrieval and locate suitable references from the database to conduct generation. Then, the motion-language generative model is trained with prompt-based question-and-answer tasks designed for different motion-relevant problems. We incorporate the retrieved samples into the prompt, and then perform instruction tuning of the motion-language model, to learn from task feedback and produce promising results with the help of fine-grained multi-modal retrieval. Extensive experiments validate the efficacy of ReMoGPT, showcasing its superiority over existing state-of-the-art methods. The framework performs well on multiple motion tasks, including motion retrieval, generation, and captioning.
BibTeX
@inproceedings{yu2025remogpt,
  author = {Qing Yu and Mikihiro Tanaka and Kent Fujiwara},
  title = {{ReMoGPT: Part-Level Retrieval-Augmented Motion-Language Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i9.33044},
}

Loading related content...

Figure