PoseLLaVA: Pose Centric Multimodal LLM for Fine-Grained 3D Pose Manipulation

Dong Feng, Ping Guo, Encheng Peng, Mingmin Zhu, Wenhao Yu, Peng WangAAAI 2025 · DOI

Introduction

While Large Language Models (LLMs) have demonstrated remarkable capabilities in understanding and generating text and images, they often struggle with the geometric precision required for fine-grained 3D human pose manipulation. Current vision-language models can handle coarse descriptions like "a person walking," but they lack the structural understanding to execute specific commands such as "adjust the right arm to be horizontal" or "put both hands in front of the stomach."

PoseLLaVA addresses these challenges by treating 3D human pose as a first-class modality. By integrating a dedicated pose encoder-decoder architecture and a novel fine-grained dataset, PoseLLaVA achieves state-of-the-art results in 3D pose estimation, generation, and language-driven adjustment.

Overview of PoseLLaVA capabilities in pose estimation, adjustment, generation, and image editing.

Methodology

PoseLLaVA builds upon the LLaVA framework but introduces a pose-centric architecture designed for geometric precision. The model represents human poses using the SMPL model (specifically 6D rotations for 24 joints) and utilizes a CLIP-ViT vision encoder alongside a Mistral-7B backbone.

The training process is divided into three distinct stages to ensure robust alignment across modalities:

  1. Stage 1: Pose-Image Alignment: Using a contrastive learning approach (SigLip), the model aligns pose features with visual features in a shared latent space. A vision resampler maps local visual tokens to pose tokens.
  2. Stage 2: Pose-Text Alignment: The LLM is pre-trained on pose generation tasks to bridge the gap between natural language descriptions and mathematical 3D pose representations.
  3. Stage 3: Multi-Task Instruction Tuning: The entire system is fine-tuned end-to-end to handle three primary tasks: Pose Estimation (Image to Pose), Pose Generation (Text to Pose), and Pose Adjustment (Initial Pose + Text to Target Pose).
The three-stage training strategy of PoseLLaVA, including alignment and instruction tuning.

Datasets and the PosePart Innovation

To overcome the limitations of existing datasets, which often feature redundant descriptions and complex multi-limb changes, the authors introduced PosePart. This new dataset consists of 135,000 samples focusing on single body part changes. By isolating movements (e.g., just the left arm), the model learns cleaner mappings between specific language instructions and geometric changes. Descriptions were further refined using GLM-4 to ensure they are natural and concise.

In addition to PosePart, the model was trained and evaluated on standard benchmarks including Human3.6M, 3DPW, PoseScript, and PoseFix.

Key Results and Insights

PoseLLaVA demonstrates significant quantitative and qualitative improvements over existing methods:

  • Pose Adjustment: On the PoseFix benchmark, the model reduced the Mean Per-Joint Position Error (MPJPE) by 45% compared to previous state-of-the-art models.
  • Pose Generation: It achieved a 20% reduction in MPJPE compared to methods like PoseScript and ChatPose.
  • Pose Estimation: PoseLLaVA outperformed recent multimodal LLMs on the Human3.6M benchmark, even without extensive data augmentation.
  • Synergy with Diffusion: One of the most powerful applications is using PoseLLaVA as a "geometric brain" for diffusion models. By calculating exact 3D coordinates first, it enables pixel-perfect image editing via tools like ControlNet.

Conclusion and Future Work

PoseLLaVA represents a significant step forward in fine-grained 3D pose control within the LLM framework. By treating pose as a language-like modality and utilizing "atomized" training data, it achieves unprecedented precision. Future work will focus on expanding the model to handle multi-person interactions and more complex, cluttered real-world environments, which are critical for applications in robotics and digital media production.

Summary updated

BibTeX
@inproceedings{feng2025posellava,
  author = {Dong Feng and Ping Guo and Encheng Peng and Mingmin Zhu and Wenhao Yu and Peng Wang},
  title = {{PoseLLaVA: Pose Centric Multimodal LLM for Fine-Grained 3D Pose Manipulation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i3.32302},
}
Figure