InsPose: Instance-Aware Networks for Single-Stage Multi-Person Pose Estimation

Dahu Shi, Xing Wei, Xiaodong Yu, Wenming Tan, Ye Ren, Shiliang Pu

TL;DR

This paper introduces InsPose, a novel single-stage framework for multi-person pose estimation that dynamically adapts network parameters for each detected person. This instance-aware approach significantly improves accuracy and efficiency compared to existing single-stage methods and offers a competitive trade-off with state-of-the-art two-stage methods.

What Problem Does This Paper Address?

Multi-person pose estimation, the task of identifying all individuals in an image and pinpointing their key body joints, is a fundamental problem with applications ranging from motion recognition to athletic training. Existing methods typically fall into two categories:

  • Top-down methods: These first detect each person with a bounding box and then perform single-person pose estimation within that box. While often accurate, they suffer from high computational redundancy and are not end-to-end trainable.
  • Bottom-up methods: These detect all keypoints in an image first and then group them into individual poses. They are more efficient but often rely on heuristic grouping strategies that limit end-to-end learning and can be less accurate.

Single-stage methods aim to simplify this by performing detection and pose estimation in one go, offering end-to-end trainability and a more compact pipeline. However, previous single-stage methods have struggled with performance limitations, particularly in accurately regressing diverse full-body poses from a single feature representation. They often require complex heuristic designs or struggle to capture the fine-grained pose variations of multiple individuals.

Methods

InsPose tackles these challenges by introducing an instance-aware dynamic network approach. The core idea is to move away from fixed network parameters for all instances and instead generate adaptive parameters for each individual person.

  1. Instance-Aware Dynamic Networks (KP-Nets): Instead of a single, static keypoint network, InsPose dynamically generates a set of "KP-Nets" for each detected instance. These KP-Nets are essentially small, lightweight convolutional neural networks (CNNs) whose parameters (weights and biases) are predicted on-the-fly based on the input features. This allows each KP-Net to learn parameters specifically tailored to the pose of a particular person, enhancing its capacity and adaptability. The KP-Nets themselves are compact, consisting of just three 1x1 convolutional layers.
  2. Disk Offset Estimation: To address the discretization error introduced by down-sampling feature maps (a common practice in CNNs for efficiency), InsPose incorporates a "disk offset prediction branch." This branch predicts an instance-agnostic local offset for each keypoint. This offset is then used to refine the predicted keypoint location, improving localization accuracy, especially for smaller instances.
  3. Network Architecture: The framework builds upon the anchor-free object detector FCOS, utilizing a ResNet or HRNet backbone with a Feature Pyramid Network (FPN) for multi-scale feature extraction. The instance-aware KP-Nets and the disk offset branch operate on these FPN features.
  4. Multi-Task Learning: For improved training, InsPose also employs an auxiliary heatmap prediction task, which is only used during training and discarded during inference.

Datasets

The paper primarily uses the MS-COCO dataset for both training and evaluation. Specifically, they use the train2017 split for training and the val2017 split for ablation studies. Main results are reported on the test-dev split.

Key Insights and Ideas

  • Dynamic Parameter Generation is Key: The most significant insight is that adapting network parameters per instance is crucial for robust multi-person pose estimation in a single-stage setting. This allows the network to capture the diverse and complex variations in human poses more effectively than a one-size-fits-all approach.
  • Compactness and Efficiency: The proposed KP-Nets are intentionally designed to be very small (three 1x1 convolutions), ensuring that the dynamic generation of parameters doesn't lead to prohibitive computational overhead. This is a critical factor for achieving a good balance between accuracy and efficiency.
  • Disk Offset for Refinement: The disk offset prediction is a clever way to mitigate a common issue in dense prediction tasks (like pose estimation) arising from down-sampling. It provides a simple yet effective mechanism to recover lost localization precision.
  • Single-Stage Advantage: By eliminating the need for separate person detectors or complex grouping heuristics, InsPose offers a streamlined, end-to-end trainable pipeline that is inherently more efficient.

Key Results

InsPose demonstrates strong performance on the MS-COCO benchmark:

  • State-of-the-Art Single-Stage Performance: InsPose significantly outperforms existing single-stage methods. For example, with a ResNet-50 backbone, it achieves 65.4% AP, outperforming DirectPose by +3.2%. With an HRNet-w32 backbone, it reaches 71.0% AP, surpassing Point-Set Anchors.
  • Competitive with Two-Stage Methods: The framework achieves a better trade-off between accuracy and efficiency compared to state-of-the-art two-stage approaches. While top-down methods might achieve slightly higher AP, InsPose is considerably faster. For instance, InsPose with HRNet-w32 infers at 0.15s per image, significantly faster than HRNet-w32 used in top-down methods (>2.34s).
  • Ablation Studies:
    • The "disk offset" module alone improves AP from 60.3% to 62.1%.
  • Adding the auxiliary heatmap task further boosts performance to 63.1% AP.
  • The depth of the KP-Net is explored, with a depth of 3 layers showing optimal performance.

Limitations and Future Work

  • Performance Gap with Top-Down: While InsPose is competitive, it still lags slightly behind the absolute best-performing top-down methods. The paper notes that top-down methods can mitigate scale variance by cropping and resizing, a strategy not directly employed in their single-stage approach.
  • Bounding Box Detection Integration: The paper mentions that InsPose can be easily extended to perform bounding box detection simultaneously. While they show competitive results (52.5% box AP), the keypoint performance slightly drops due to label assignment differences. This suggests potential for further optimization in joint detection and pose estimation.
  • Future Directions: The authors suggest that their instance-aware dynamic network approach could be beneficial for other dense prediction tasks. The ability to integrate bounding box detection also opens avenues for more comprehensive human understanding systems.

Summary updated

BibTeX
@inproceedings{shi2021inspose,
  author = {Dahu Shi and Xing Wei and Xiaodong Yu and Wenming Tan and Ye Ren and Shiliang Pu},
  title = {{InsPose: Instance-Aware Networks for Single-Stage Multi-Person Pose Estimation}},
  booktitle = {ACM Multimedia},
  year = {2021},
  doi = {10.1145/3474085.3475447},
}

Loading related content...

Figure