Introduction
While Large Language Models (LLMs) have demonstrated remarkable capabilities in understanding and generating text and images, they often struggle with the geometric precision required for fine-grained 3D human pose manipulation. Current vision-language models can handle coarse descriptions like "a person walking," but they lack the structural understanding to execute specific commands such as "adjust the right arm to be horizontal" or "put both hands in front of the stomach."
PoseLLaVA addresses these challenges by treating 3D human pose as a first-class modality. By integrating a dedicated pose encoder-decoder architecture and a novel fine-grained dataset, PoseLLaVA achieves state-of-the-art results in 3D pose estimation, generation, and language-driven adjustment.
Methodology
PoseLLaVA builds upon the LLaVA framework but introduces a pose-centric architecture designed for geometric precision. The model represents human poses using the SMPL model (specifically 6D rotations for 24 joints) and utilizes a CLIP-ViT vision encoder alongside a Mistral-7B backbone.
The training process is divided into three distinct stages to ensure robust alignment across modalities:
- Stage 1: Pose-Image Alignment: Using a contrastive learning approach (SigLip), the model aligns pose features with visual features in a shared latent space. A vision resampler maps local visual tokens to pose tokens.
- Stage 2: Pose-Text Alignment: The LLM is pre-trained on pose generation tasks to bridge the gap between natural language descriptions and mathematical 3D pose representations.
- Stage 3: Multi-Task Instruction Tuning: The entire system is fine-tuned end-to-end to handle three primary tasks: Pose Estimation (Image to Pose), Pose Generation (Text to Pose), and Pose Adjustment (Initial Pose + Text to Target Pose).
Datasets and the PosePart Innovation
To overcome the limitations of existing datasets, which often feature redundant descriptions and complex multi-limb changes, the authors introduced PosePart. This new dataset consists of 135,000 samples focusing on single body part changes. By isolating movements (e.g., just the left arm), the model learns cleaner mappings between specific language instructions and geometric changes. Descriptions were further refined using GLM-4 to ensure they are natural and concise.
In addition to PosePart, the model was trained and evaluated on standard benchmarks including Human3.6M, 3DPW, PoseScript, and PoseFix.
Key Results and Insights
PoseLLaVA demonstrates significant quantitative and qualitative improvements over existing methods:
- Pose Adjustment: On the PoseFix benchmark, the model reduced the Mean Per-Joint Position Error (MPJPE) by 45% compared to previous state-of-the-art models.
- Pose Generation: It achieved a 20% reduction in MPJPE compared to methods like PoseScript and ChatPose.
- Pose Estimation: PoseLLaVA outperformed recent multimodal LLMs on the Human3.6M benchmark, even without extensive data augmentation.
- Synergy with Diffusion: One of the most powerful applications is using PoseLLaVA as a "geometric brain" for diffusion models. By calculating exact 3D coordinates first, it enables pixel-perfect image editing via tools like ControlNet.
Conclusion and Future Work
PoseLLaVA represents a significant step forward in fine-grained 3D pose control within the LLM framework. By treating pose as a language-like modality and utilizing "atomized" training data, it achieves unprecedented precision. Future work will focus on expanding the model to handle multi-person interactions and more complex, cluttered real-world environments, which are critical for applications in robotics and digital media production.
Summary updated