HPnet: Hybrid Parallel Network for Human Pose Estimation

Haoran Li, Hongxun Yao, Yuxin HouSensors 2023 · DOI

TL;DR

This paper introduces HPnet, a novel hybrid parallel network for human pose estimation that addresses the conflict between convolutional and self-attention modules in existing sequential hybrid models. By processing features in parallel and using a cross-branches attention mechanism, HPnet effectively leverages the complementary strengths of both approaches, leading to improved accuracy and consistent performance in localizing human keypoints.

What Problem Does This Paper Address?

Human pose estimation, a crucial task in computer vision with applications ranging from action recognition to human-computer interaction, aims to pinpoint specific anatomical keypoints on a person in an image. While deep convolutional neural networks (CNNs) have achieved significant progress, they struggle with the inherent variability in human poses due to their local receptive fields. Transformer models, with their self-attention mechanisms, excel at capturing long-range dependencies, but can sometimes sacrifice the precise localization capabilities of CNNs. Existing hybrid models often stack self-attention modules after CNNs, leading to a "mutual conflict" where one module's strengths can dominate and degrade the other's performance, particularly affecting high-precision keypoint localization. This paper tackles this conflict by proposing a parallel architecture that allows both modules to operate independently and complement each other.

Methods

The core innovation of HPnet lies in its hybrid parallel architecture. Unlike sequential models that process features one after another, HPnet processes features through two parallel branches:

  • Self-Attention Branch: This branch utilizes a standard transformer encoder architecture to capture long-range dependencies and model global semantic information. To manage computational cost, a down-sampling module is incorporated.
  • Convolutional Branch: This branch employs a modified ResNet-like structure to extract locally sensitive features and maintain high-precision localization.

Crucially, HPnet introduces a Cross-Branches Attention (CBA) module to fuse the features from these two parallel branches. This module acts as a soft gate, allowing each branch to selectively influence the other, thereby mitigating the semantic conflict and enabling a more effective complementary fusion of features. This contrasts with simple addition, which can disrupt the individual properties of the features. The model then uses a standard three-layer transpose convolutional operator as an output head to generate heatmaps for keypoint prediction.

Datasets

The paper evaluates HPnet on two widely used datasets for human pose estimation:

  • COCO (Common Objects in Context): This is a large-scale dataset featuring 118k training images and 41k testing images, with 100k individuals annotated with 17 keypoints. HPnet is trained on the training set and evaluated using the Object Keypoints Similarity (OKS) metric on validation and test-dev sets.
  • MPII (Multimodal Human Pose Dataset): This dataset is used for evaluating performance on a different configuration of human poses (16 joints without facial landmarks). The PCKh metric is employed for evaluation on this dataset.

Key Insights and Ideas

  • Parallelism is Key to Avoiding Conflict: The paper strongly argues that a parallel architecture, rather than a sequential one, is essential for effectively combining the strengths of convolutional and self-attention modules without one dominating the other.
  • Cross-Branches Attention for Harmonious Fusion: The proposed CBA module is a critical component, enabling a more nuanced fusion of features by learning to gate and integrate information from both branches, thus resolving the inherent conflict.
  • Complementary Strengths for High-Precision Localization: The parallel structure allows the self-attention branch to excel at understanding global context and semantic relationships, while the convolutional branch focuses on fine-grained local details, leading to more accurate keypoint localization, especially for challenging joints.
  • Position Embedding Can Be Detrimental in Hybrid Parallel Models: Counterintuitively, the authors found that adding position embedding to the self-attention branch in their parallel architecture decreased performance. They attribute this to the convolution branch already modeling position-sensitive information, and the injected positional information creating a conflict.

Key Results

HPnet demonstrates competitive performance on both COCO and MPII datasets:

  • COCO: HPnet achieves 75.6% AP on the validation set and 75.4% AP on the test-dev set, comparable to state-of-the-art models. Notably, it shows improvements in AP75 (high-precision localization) compared to sequential hybrid models like TransPose.
  • MPII: HPnet achieves an overall PCKh of 91.8% on the validation set, surpassing methods like TokenPose and HRNet, with significant improvements on challenging joints like elbows and ankles.
  • Ablation Studies: Experiments confirm the effectiveness of the self-attention branch and the CBA module, showing that adding self-attention blocks improves AP, and the CBA module further enhances performance, particularly AP75.

Limitations and Future Work

  • Small and Blurred Instances: The paper acknowledges that small and blurred human instances remain a challenge, degrading performance due to lost details and distinctness.
  • Crowded Scenarios: In crowded scenes, occluded human instances can still lead to errors, where the model struggles to infer invisible joints without appearance information.
  • Jitter Errors: While HPnet mitigates many errors, "Jitter" (where OKS is between 0.5 and 0.85) remains a notable error type. The authors suggest that higher-resolution input images or improved heatmap representations could further alleviate this.
  • Future Work: The authors plan to investigate further ways to improve performance on challenging cases like small instances and occlusion, and explore leveraging larger models and richer structural information of human poses for even higher-precision keypoint localization.

Summary updated

BibTeX
@article{li2023hpnet,
  author = {Haoran Li and Hongxun Yao and Yuxin Hou},
  title = {{HPnet: Hybrid Parallel Network for Human Pose Estimation}},
  journal = {Sensors},
  year = {2023},
  doi = {10.3390/s23094425},
}

Loading related content...

Figure