LAMP: Leveraging Language Prompts for Multi-Person Pose Estimation

Shengnan Hu, Ce Zheng, Zixiang Zhou, Chen Chen, Gita Sukthankar

Abstract

Multi-person pose estimation (MPPE) in crowded and occluded environments remains a significant challenge for computer vision. Traditional single-stage methods often lack the semantic awareness required to distinguish between overlapping individuals or to localize hidden joints. This paper introduces LAMP (Language Assisted Multi-person Pose estimation), a framework that incorporates semantic knowledge from pre-trained language models like CLIP to enhance pose estimation. By aligning visual features with descriptive text prompts at the instance, joint, and pixel levels, LAMP achieves state-of-the-art performance on challenging benchmarks, demonstrating superior robustness in complex scenes.

Introduction

The goal of multi-person pose estimation is to identify all individuals in an image and accurately localize their key anatomical joints. While current models perform reliably in clear, sparse conditions, their performance degrades significantly in "crowded" scenarios where individuals overlap or body parts are occluded by objects or other people.

Existing approaches generally fall into two categories: top-down and bottom-up. Top-down methods rely on an initial person detection box, which can be inaccurate in dense crowds. Bottom-up methods detect all joints and attempt to group them, a process that often fails when people are in close proximity. LAMP proposes a single-stage estimation approach that overcomes these limitations by transforming pose estimation from a pure pixel-regression task into a multi-modal alignment task, leveraging the semantic "sanity check" provided by natural language.

Methodology

LAMP utilizes the pre-trained knowledge of CLIP (Contrastive Language-Image Pretraining) to provide a semantic prior for the visual model. The framework consists of three primary components designed to resolve ambiguity in crowded scenes:

1. Instance Prompts with Attribute Injection

To decouple overlapping individuals, LAMP generates unique text descriptions for every person in a scene. Using templates such as "A {size} {occluded} person on the {location}," the model assigns a semantic identity to each instance (e.g., "A close occluded person on the bottom right"). This allows the model to differentiate between individuals even when their visual features are heavily intertwined.

2. Joint Prompts

LAMP employs text labels for specific joints (e.g., "left ankle," "nose"). Since CLIP understands the relationship between these linguistic terms and visual concepts, these prompts act as a semantic prior. This guides the model toward the correct anatomical region, even when the joint itself is not visible to the camera.

3. Multi-level Alignment

The framework ensures consistency between language and vision through three levels of alignment:

  • Instance-level Alignment: Maximizes the correlation between a person’s visual features and their specific descriptive text prompt.
  • Keypoint-level Alignment: Uses contrastive learning to ensure that the feature at a specific joint location matches the embedding of that joint's name.
  • Pixel-level Alignment: Maps joint descriptions directly to heatmap pixels, refining the final localization and significantly improving the accuracy of dense regression.

Experimental Results

The authors evaluated LAMP on two benchmarks specifically curated for high-difficulty scenarios: OCHuman (extremely challenging occlusions) and CrowdPose (dense crowds).

Performance on OCHuman

LAMP achieved 58.8% AP (Average Precision) on the OCHuman dataset. This result outperforms the previous leading single-stage model, CID, by 1.3%. Notably, it significantly surpasses traditional top-down methods like Mask R-CNN, which only achieved 20.2% AP on this challenging set.

Performance on CrowdPose

On the CrowdPose benchmark, LAMP reached 71.4% AP, setting a new standard for efficiency and accuracy in single-stage models. Qualitative analysis confirms that LAMP is uniquely capable of "seeing through" overlaps, correctly assigning limbs to the appropriate person even in complex, intertwined poses.

Discussion and Insights

Several key findings emerged from the development of LAMP:

  • Language as a Global Context: In ambiguous visual environments, language provides a global context that prevents the model from making anatomically impossible errors.
  • The Necessity of Frozen Encoders: Keeping the CLIP text encoder frozen during training was essential. This preserved the model's general linguistic knowledge, preventing it from over-fitting to the specific pose estimation task at the expense of semantic breadth.
  • The Impact of Pixel-Alignment: Ablation studies indicated that pixel-level alignment—connecting joint names directly to heatmap regression—provided the most significant boost in localization accuracy.

Conclusion and Future Work

LAMP demonstrates that integrating language prompts into the pose estimation pipeline provides the semantic awareness necessary to navigate crowded and occluded scenes. While the current model relies on fixed templates for prompt generation, future iterations may leverage Large Language Models (LLMs) to generate more descriptive, natural-language context. Additionally, the authors plan to extend this multi-modal approach to 3D human mesh recovery and social interaction recognition, potentially enhancing the capabilities of social robots in real-world environments.

Summary updated

BibTeX
@inproceedings{hu2023lamp,
  author = {Shengnan Hu and Ce Zheng and Zixiang Zhou and Chen Chen and Gita Sukthankar},
  title = {{LAMP: Leveraging Language Prompts for Multi-Person Pose Estimation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  doi = {10.1109/iros55552.2023.10341430},
}
Figure